This Letter presents a spectral-domain, polarization-sensitive optical coherence tomography (PS-OCT) system, where the light collection from the two arms of the interferometer is performed exclusively using single-mode fibers and couplers, and the two orthogonal polarization components are sequentially detected by a single line camera. Retardance measurements can be affected by polarimetric effects because of fiber birefringence and diattenuation in fiber couplers. This configuration bypasses such issues by performing polarization selection before the collection fiber through the combination of a polarization rotator and a linear polarizer. Retardance calibration is achieved with a Berek compensator. Similar net retardance maps of a birefringent phantom are obtained for two different settings of induced fiber birefringence, effectively demonstrating the tolerance of the configuration to fiber-based disturbances.