Here we combined experiments and theory to study the optical properties of a plasmonic cavity consisting of a perforated metal film and a flat metal sheet separated by a semiconductor spacer. Three different types of optical modes are clearly identified-the propagating and localized surface plasmons on the perforated metal film and the Fabry-Perot modes inside the cavity. Interactions among them lead to a series of hybridized eigenmodes exhibiting excellent spectral tunability and spatially distinct field distributions, making the system particularly suitable for multicolor infrared light detections. As an example, we design a two-color detector protocol with calculated photon absorption efficiencies enhanced by more than 20 times at both colors, reaching ~42.8% at f1 = 20.0THz (15μm in wavelength) and ~46.2% at f2 = 29.5THz (~10.2μm) for a 1μm total thickness of sandwiched quantum wells.