Plasmonic metamaterials have overcome fundamental limitations in conventional optics by their capability to engineer material resonances and dispersions at will, holding great promise for sensing applications. Recent demonstrations of metamaterial sensors, however, have mainly relied on their resonant nature for strong optical interactions with molecules, but few examples fully exploit their functionality to manipulate the polarization of light. Here, we present cross-polarized surface-enhanced infrared absorption (SEIRA) by the Fano-resonant asymmetric metamaterial allowing for strong background suppression as well as significant field enhancement. The metamaterial is designed to exhibit the controlled Fano resonance with the cross-polarization conversion property at 1730 cm-1, which spectrally overlaps with the C=O vibrational mode. In the cross-polarized SEIRA measurement, the C=O mode of poly(methyl methacrylate) molecules is clearly observed as a distinct dip within a Fano-resonant transmission peak of the metamaterial. The vibrational signal contrast is then improved based on the cross-polarized detection scheme where only the light interacting with the metamaterial-molecular coupled system is detected by totally eliminating the unwanted background light. Our metamaterial approach achieves the zeptomole sensitivity with a large signal-to-noise ratio in the far-field measurement, paving the way toward the realization of ultrasensitive IR inspection technologies.