The periodic nonlinearity that arises from nonideal laser sources and imperfections of optical components limits the accuracy of displacement measurements in heterodyne interferometry at the nanometer level. An analytical approach to investigating the nonlinearity is presented. Frequency mixing, polarization mixing, polarization-frequency mixing, and ghost reflections are all included in this investigation. A general form for the measurement signal, including that of the distortions, is given. The analytical approach is also applicable to homodyne interferometry.