White light interferometry is a well-established surface recovery technique. In this paper, a white light signal processing algorithm based on phase error compensation using spectrum selection is proposed. The derived nonlinear phase distribution from the correlogram is modeled as the combination of random errors and systemic deviations. By developing a new, to the best of our knowledge, recovery algorithm, the phase noise can be separated from the linear map and significantly attenuated. Based on the proposed algorithm, the spectrum features of white light LEDs and halogen lamps are investigated in detail. The inner products defined by three selected points are employed to generate a coefficient to evaluate the linearity of an unwrapped phase map within a certain spectrum region. The optimal spectrum range corresponding to the best measurement performance can then be located where the coefficient approximates 1 and the spectrum energy stays relatively high. The simulations are carried out under different levels of SNR and scan step noises, which show that the new method can effectively reduce additional disturbance from the recovered topography. In experiments, the system with the proposed method is first calibrated by a step height standard (VLSI, 182.7±2.0nm) with the repeatability of 0.44 nm. A silicon wafer and three roughness standards are also tested to further verify the robustness of the new method.