Frequency modulation spectroscopy (FMS) not only can be used to simultaneously measure the absorption and dispersion of atoms and molecules, but is the key technology of the noise immunity cavity enhanced optical heterodyne molecular spectroscopy (NICE-OHMS). The optical devices or the instability of output light polarization of the laser source will induce the residual amplitude modulation (RAM) in the FMS. RAM greatly limits the FMS technology application in trace gas detection, so the research on the RAM characteristics in the FMS has very important significance. Firstly, the lineshape of FMS without absorption was analyzed, and the impact factors on the RAM were acquired, then the influence of input and output polarization direction and electro-optical modulation (EOM) temperature was measured, respectively. They all have linear relationship with the RAM. The results verify the theoretical analysis and provide the basis for reducing the RAM and other related working.