Combined with the advantages of photoelastic modulator (PEM) ultra-high-speed modulation, this paper presents a method of ultra-high-speed spectropolarimeter based on PEM. The method provides the necessary measuring instruments for ultra-high-speed polarization spectroscopy. The main idea of this method is that an intensity modulator consisting of two retarders is placed before the PEM. The incident light under test goes through two retarders to the PEM. The interference signals are obtained by the PEM modulation. The different Stokes element interference signals are modulated by the PEM at different positions of the optical path difference. This method realizes the separation of Stokes element interference signals. The interference signals corresponding to each element are extracted, and the incident light Stokes element spectra can be obtained from the Fourier transforms of the interference signals. The modulation frequency of the PEM is high (tens to hundreds of kilohertz), so this method can realize ultra-high-speed full polarization spectroscopy. A prototype ultra-high-speed spectropolarimeter based on PEM was designed and tested. If the single-sided Fourier transformation is used, the single-sided interferogram scanning time is approximately 5 μs (i.e., the prototype is capable of scanning 20,000 interferograms per second). Polychromatic light polarization spectroscopy is measured by the prototype. The experimental results show that the average error of the prototype is less than 0.03.