In this investigation, we describe polarized low coherence scanning interferometry (PLCSI) to enhance the measurement speed based on the spatial phase shifting technique by using a polarized CMOS camera. In every scanning step, the visibility of the correlogram can be directly extracted by spatial phase shifting. PLCSI does not need any scanning conditions such as a scanning step size smaller than that determined by the Nyquist sampling limit and equidistant scanning step, which restrict the measurement speed of the typical low coherence scanning interferometry (LCSI). The measurement data can also be significantly reduced due to the larger scanning step size. PLCSI can be comparable to confocal scanning microscopy in the view of monitoring visibilities. In the experiments, three types of specimens such as a plane mirror, a concave mirror, and a step height specimen were measured by PLCSI with various scanning step sizes, and it was confirmed that the surface profiles were successfully reconstructed. Moreover, the compensation technique of the surface profile, precisely determined by the phase information, was also discussed.