We propose an accurate and rapid azimuth calibration method for polarizing elements in ellipsometry. Over 200 calibrations were achieved simultaneously at different wavelength points in a spectral range of 550-650 nm without any calibrated element. The azimuth of the polarizer was determined from the differential spectral analysis on the ellipse azimuth of reflected light. The information of the ellipse azimuth is experimentally acquired in the spectral range by a rotating polarizing element and a spectrometer. The presented method was performed and verified with Si and Au bulk, respectively, showing reliability and feasibility for efficient and reliable calibration in ellipsometry.