We probe the indistinguishability of photons emitted by a semiconductor quantum dot (QD) via time- and temperature-dependent two-photon interference (TPI) experiments. An increase in temporal separation between consecutive photon emission events reveals a decrease in TPI visibility on a nanosecond time scale, theoretically described by a non-Markovian noise process in agreement with fluctuating charge traps in the QD's vicinity. Phonon-induced pure dephasing results in a decrease in TPI visibility from (96±4)% at 10 K to a vanishing visibility at 40 K. In contrast to Michelson-type measurements, our experiments provide direct access to the time-dependent coherence of a quantum emitter on a nanosecond time scale.