Based on Fourier domain estimation, a novel self-calibration phase-shifting algorithm, named Mid-Band Spatial Spectrum Matching (MSSM), is proposed to achieve phase retrieval from a small amount of phase-shifting interferograms containing very few fringes (defined as ultra-sparse fringe pattern(USFP)), which is still a difficult problem for optical interferometry. Both simulation and experimental results demonstrate that the proposed MSSM algorithm can accurately and rapidly achieve the phase distribution encoded in USFP while other current self-calibration algorithms fail, and this will supply a powerful tool to extend the application of phase-shifting interferometry.