Optical cooling of a YLF:Yb single crystal to 87 K, well below the minimum achievable temperature predicted from existing theory, has been observed. This discrepancy between theory and data has motivated us to revisit the current model of optical refrigeration, in particular the critical role of parasitic background absorption. Challenging experiments that measured the cooling efficiency as a function of temperature reveal that the background absorption coefficient decreases with temperature, resulting in a significant enhancement of the cooling efficiency at cryogenic temperatures. These discoveries emphasize the high sensitivity of optical cooling to impurity-mediated processes and show the necessity of formulating a cooling model that includes the temperature dependence of the background absorption. To properly characterize the cooling properties of any sample, it is necessary to measure its low-temperature performance.