Frequency scanning interferometry (FSI) is a promising technique for absolute distance measurement and has been demonstrated in many industrial applications. However, in practice, the measurement precision is limited and sensitive to the variations of the measured distance while sweeping the optical frequency of the laser. The induced errors would be amplified by hundreds of times due to the magnification effect. In this paper, an incremental interferometer was established on the basic scheme of the FSI system for monitoring the variations of distance. The compensation could be achieved by multiplying the heterodyne signals from monitor and measurement interferometer without complex and time-costing data processing. The system performance has been verified by experiments for different kinds of vibrating targets. Finally, after compensation by suppression of the magnification effects, a measurement precision of 4.26 μm has been achieved in a range of 10 m.