Nonlinear frequency conversion provides an elegant method to detect photons in a spectral range which differs from the pump wavelength, making it highly attractive for photons with inherently low energy. Aside from the intensity of the light, represented by the number of photons, their phase provides important information and enables a plethora of applications. We present a phase-sensitive measurement method in the terahertz spectral range by only detecting visible light. Using the optical interference of frequency-converted photons and leftover pump photons of the involved ultrashort pulses, fast determination of layer-thicknesses is demonstrated. The new method enables phase-resolved detection of terahertz pulses using standard sCMOS equipment while achieving sample measurement times of less than one second with a precision error of less than 0.6%.