Optical bistability typically occurs only when the optical thickness in the device or the input light power is unfavorably large. Here we show that, for a class of plasmonic metamaterials consisting of ultrathin holey metallic plates filled with nonlinear materials, the optical bistability can occur with an ultralow excitation power. We present a realistic design working at 0.2 THz and perform full-wave simulations to quantitatively study its optical bistability properties. An analytical model is developed to explain the inherent physics and provides a general design guideline for future development.