We demonstrate an environment-noise-free optical heterodyne interferometer (OHI) based on a double-pass acousto-optic frequency shifter (AOFS) for phase retardation measurements. The OHI generates intermediate-frequency (IF) heterodyne beat signals for each orthogonal linear polarization mode of a birefringence sample. The phase differences of the IF signals are compared by using a wide-bandwidth lock-in amplifier. This scheme provides 20 dB rejection of common-mode environmental disturbances. Measurements of the half-wave voltage of an electro-optics modulator and the electro-optic coefficient of an LiNbO3 plate are demonstrated as application examples.