Gas-phase IR spectra of carbon dioxide and methane are nowadays well understood, as a consequence of their pivotal roles in atmospheric- and astrochemistry. However, once those molecules are trapped in noble gas matrices, their spectroscopic properties become difficult to conceptualize. Still, such spectra provide valuable insights into the vibrational structure. In this study, we combine new matrix-isolation infrared (MI-IR) spectra at 6 K in argon and neon with in vacuo anharmonic spectra computed by vibrational self-consistent field (VSCF) and vibrational configuration interaction (VCI). The aim is to separate anharmonicity from matrix effects in the mid-infrared spectra of 12C16O2, 12CH4, and 12CD4. The accurate description of anharmonic potential energy surfaces including mode-coupling allows to reproduce gas-phase data with deviations of below 3 cm-1. Consequently, the remaining difference between MI-IR and VSCF/VCI can be attributed to matrix effects. Frequency shifts and splitting patterns turn out to be unsystematic and dependent on the particular combination of analyte and noble gas. While in the case of neon matrices these effects are small, they are pronounced in xenon, krypton, and argon matrices. Our strategy allows us to suggest that methane rotates in neon matrices - in contrast to previous reports.