We demonstrate that Casimir-Polder potentials can be entirely independent of temperature even when allowing for the relevant thermal photon numbers to become large. This statement holds for potentials that are due to low-energy transitions of a molecule placed near a plane metal surface whose plasma frequency is much larger than any atomic resonance frequencies. For a molecule in an energy eigenstate, the temperature independence is a consequence of strong cancellations between nonresonant potential components and those due to evanescent waves. For a molecule with a single dominant transition in a thermal state, upward and downward transitions combine to form a temperature-independent potential. The results are contrasted with the case of an atom whose potential exhibits a regime of linear temperature dependence. Contact with the Casimir force between a weakly dielectric and a metallic plate is made.