Weak material nonlinearity at optical frequencies poses a serious hurdle to realizing optical bistability at low optical powers, which is a critical component for digital optical computing. In this paper, we explore the cavity enhancement of the second-order optical nonlinearity in order to determine the feasibility of few photon optical bistability. Starting from a quantum optical formalism of a doubly resonant cavity (required to meet the condition of phase matching), we derive a dynamic classical model of a cavity that is bistable at the fundamental mode. We analyze the optical energy and the switching speed as a function of the cavity quality factors and mode volumes and identify the regime where only ten's of photons are required to perform the switching. An unusual trend in the switching speed is also observed, where the speed monotonically decreases as the cavity linewidth increases. This is ascribed to the increase in the switching gain with increasing cavity linewidth.