Herein we propose a method to mitigate a position mismatch problem for a spectral-domain polarization-sensitive optical coherence tomography (SD-PS-OCT) system that uses a single line-scan detection scheme. A single detector-based PS-OCT detects two orthogonal polarization components as two adjacent A-scan signals in turns. Thus, two adjacent A-scan signals are not scattered at a fixed point in time (position mismatch problem), resulting in uncorrelated signals between them. To achieve sequential detection of simultaneously scattered light, a buffering single-mode fiber was connected to one of the two ports coming out of the optical switch, provided a proper time delay. A single-mode optical fiber of 2.69 km in length was used to buffer, and its length was determined by a frame rate of the spectrometer used as a detector. With the proposed SD-PS-OCT scheme, the PS-OCT system with a simple configuration, and the minimized position mismatch problem between two polarization components can be set.