Low noise detection with state-of-the-art mid-infrared (MIR) detectors (e.g., PbS, PbSe, InSb, HgCdTe) is a primary challenge owing to the intrinsic thermal background radiation of the low bandgap detector material itself. However, researchers have employed frequency upconversion based detectors (UCD), operable at room temperature, as a promising alternative to traditional direct detection schemes. UCD allows for the use of a low noise silicon-CCD/camera to improve the SNR. Using UCD, the noise contributions from the nonlinear material itself should be evaluated in order to estimate the limits of the noise-equivalent power of an UCD system. In this article, we rigorously analyze the optical power generated by frequency upconversion of the intrinsic black-body radiation in the nonlinear material itself due to the crystals residual emissivity, i.e. absorption. The thermal radiation is particularly prominent at the optical absorption edge of the nonlinear material even at room temperature. We consider a conventional periodically poled lithium niobate (PPLN) based MIR-UCD for the investigation. The UCD is designed to cover a broad spectral range, overlapping with the entire absorption edge of the PPLN (3.5 - 5 µm). Finally, an upconverted thermal radiation power of ~30 pW at room temperature (~30°C) and a maximum of ~70 pW at 120°C of the PPLN crystal are measured for a CW mixing beam of power ~60 W, supporting a good quantitative agreement with the theory. The analysis can easily be extended to other popular nonlinear conversion processes including OPO, DFG, and SHG.