White light scanning interferometry (WLSI) has been an extremely powerful technique in precision measurements. In this work, a phase noise estimation based surface recovery algorithm is proposed, which can significantly improve the measurement accuracy by decreasing the noise level in phase map coming from the systemic and environmental disturbances. The noise existed in phase map is firstly researched in spectrum domain and defined as the linear combination of complex terms at each angular wavenumber. Afterwards, based on the theoretical linearity of the phase distribution, the surface features can be redefined through establishing the function with respect to phase noise. By applying least square estimation (LSE), a spectral coefficient is defined to determine the optimal estimation of phase noise that represents the best statistical consistency with the actual case, from which a more accurate surface after removing most phase noise will then be generated. In order to testify the noise elimination ability of the proposed method, a nano-scale step height standard (9.5nm±1.0nm) is scanned, and the measurement result 9.49nm with repeatability 0.17nm is successfully achieved. Moreover, a leading edge of an aero-engine blade is also tested to investigate the potential of this method in industrial inspections. The measurement comparison with AFM is also displayed.