Epsilon-near-zero (ENZ) materials have recently been suggested as excellent candidates for constructing all-optical and electro-optical switches in the infrared. The performance of previously reported ENZ material-based optical switches, however, has been greatly hampered by the low quality- (Q-) factor of the ENZ cavity, resulting in a large required optical pump fluence or applied voltage, a large insertion loss, or a small modulation depth. Here, we propose a solution by integrating the ENZ material into a Bragg microcavity, such that the Q-factor of the coupled cavity can be dramatically enhanced. Using high-mobility Dysprosium-doped cadmium oxide (CdO) as the prototype ENZ material, we numerically show an infrared all-optical switch with its reflectance modulated from near-zero to 94% under a pump fluence of only 7 μJ cm-2, about a 59-time-reduction compared with a state-of-the-art Berreman-type cavity. Moreover, the high-Q coupled cavity can also be adopted to realize a reflective electro-optical switch. Its reflectance can be switched from near-zero to 89%, with a bias electric field well below the breakdown field of conventional gate dielectrics. The switching operation can further be extended to the transmission mode with a slightly modified cavity geometry, with its absolute transmittance modulated by 40%.