Without the limitations of fringe number and fringe shape, a dual-channel simultaneous spatial and temporal polarization (DC-SSTP) phase-shifting interferometry system is proposed to achieve rapid and accurate phase retrieval through only one-time phase-shifting procedure with unknown phase shifts. First, an arbitrary phase shifts is simultaneously introduced into two channels of DC-SSTP system by a spatial light modulator (SLM). Second, by performing the subtraction operation between each pair of phase-shifting interferograms captured in the same channel, the background deduction of interferogram can be achieved easily, so the accurate phase can be retrieved rapidly. Especially, it is found even if the fringe number in interferogram is less than one, the proposed DC-SSTP method still reveals high accuracy of phase retrieval. Both the simulation and experimental results demonstrate the outstanding performance of proposed DC-SSTP method in phase measurement.