A difference-fitting method is proposed to mitigate the low frequency drift of a phase-sensitive optical time domain reflectometry (Φ-OTDR) system caused by laser phase noise. The effective difference region for phase demodulation is theoretically analyzed and experimentally verified, which should be greater than the convolution of the spatial resolution and the length of disturbed optical fiber. Then, a median-fitting algorithm is used to obtain the phase noise of the differential region. The vibration signal of 0.2 Hz is first demodulated with the SNR of 41.79 dB on the optical fiber of 11 km. The low-frequency vibration signals of 0.05 Hz and 0.02 Hz are then successfully restored by using the difference-fitting method, which can effectively eliminate the influence of low frequency drift.