The existence of periodic nonlinear error restricts the performance of the homodyne laser vibrometer in sub-fringe amplitude vibration measurement. A homodyne laser vibrometer with nanoscale-amplitude detectability by using a liquid crystal variable retarder (LCVR) is proposed. The LCVR introduces an extra variation of optical path difference larger than the laser wavelength to acquire a full ellipse so that the nonlinearity correction parameters could be pre-extracted. The experiments showed that the nonlinear error could be well suppressed with the correction process based on the pre-extracted parameters, and the detectable minimum amplitude is less than 1 nm. In addition, measurement of vibration with the reflectivity of measured targets down to 0.048% was achieved with an automatic-gain-control module.