We present an analytical model that shows that reflection from a rough surface causes a Gaussian frequency roll-off for the spectral magnitude of a terahertz wave and reduces the signal-to-noise ratio of terahertz time-domain spectroscopy. The parameter that determines the width of the frequency roll-off is the standard deviation of the surface height distribution. Measurements of terahertz waves reflected from copper powder samples provide experimental evidence for this effect.