We present an implementation of the single-pixel imaging approach into a terahertz (THz) time-domain spectroscopy (TDS) system. We demonstrate the indirect coherent reconstruction of THz temporal waveforms at each spatial position of an object, without the need of mechanical raster-scanning. First, we exploit such temporal information to realize (far-field) time-of-flight images. In addition, as a proof of concept, we apply a typical compressive sensing algorithm to demonstrate image reconstruction with less than 50% of the total required measurements. Finally, the access to frequency domain is also demonstrated by reconstructing spectral images of an object featuring an absorption line in the THz range. The combination of single-pixel imaging with compressive sensing algorithms allows to reduce both complexity and acquisition time of current THz-TDS imaging systems.