A complex-valued transmission matrix describing a scattering medium can be constructed from a sequence of many interferometric measurements. A major challenge in such experiments is to correct for rapid phase drift of the optical system during the data acquisition process, especially when the phase drifts significantly between consecutive measurements. Therefore, a new method is presented where the exact phase drift between two measurements is characterized and corrected using a single additional measurement. This approach removes the need to continuously track the phase and significantly relaxes the phase stability requirements of the interferometer, allowing transmission matrices to be constructed in the presence of fast and erratic phase drift.