We demonstrate a method for retrieving the phase information from single-shot interference spectra obtained by dispersive Fourier transform, through which the error accumulation during phase retrieval is restrained. A Mach-Zehnder interferometer is proposed for vibration sensing with high speed. We find that relative phase trends at different time delays can be precisely retrieved to improve the signal-to-noise ratio when the time interval jitter between pulses within two arms is less than four times the pulse width. The verification experiment achieves a phase resolution of 5.3 mrad and a high-speed refresh frame rate of 51.8 MHz. Numerical simulations and experiments show that the method is effective for phase demodulation of dynamic interference spectra, and provides a reliable strategy for high-speed, precision sensing.