Temporal-spectral modes of light provide a fundamental window into the nature of quantum systems and offer a robust means for information encoding. Methods to precisely characterize the temporal-spectral state of light at the single-photon level thus play a central role in understanding single-photon sources and their applications in emerging optical quantum technologies. Here we demonstrate an optical reference-free method, which melds techniques from ultrafast metrology and single-photon spectral detection, to characterize the pulse-mode structure of heralded single photons.