The use of infrared-emitting rare-earth-doped luminescent nanoparticles as nanothermometers has attracted great attention during the last few years. The scientific community has identified rare-earth-doped luminescent nanoparticles as one of the most sensitive and versatile systems for contactless local temperature sensing in a great variety of fields, but especially in nanomedicine. Researchers are nowadays focused on the design and development of multifunctional nanothermometers with new spectral operation ranges, outstanding brightness, and enhanced sensitivities. However, no attention has been paid to the assessment of the actual reliability of the measurements provided by rare-earth-doped luminescent nanothermometers. In fact, it is assumed that they are ideal temperature sensors. Nevertheless, this is far from being true. In this work we demonstrate that the emission spectra of rare-earth-doped nanothermometers can be affected by numerous environmental and experimental factors. These include the numerical aperture of the optical elements used for their optical excitation and luminescence collection, the local concentration of nanothermometers, optical length variations, self-absorption of the luminescence by the nanothermometers themselves, and solvent optical absorption. This work concludes that rare-earth-doped luminescent nanothermometers are not as reliable as thought and, consequently, special care has to be taken when extracting temperature estimations from the variation of their emission spectra.