A general method is presented for spherical surface testing with unknown phase shifts based on a physical model of the interferometer cavity, which describes the phase shifts taking into account the rigid cavity motions and the radial imaging distortion of the interferometer. The captured interferograms are processed frame by frame with the regularized frequency-stabilizing method, so as to get the phase shifts between the frames. These phase shift data are subsequently fitted, and the initial estimations for the wavefront, direct current and interference contrast terms are calculated by the least-squares method. Specially, a simple way is proposed to find reasonable initial guess for numerical aperture (NA) of the test beam (when NA is unknown), so as to ensure the effectiveness of the above phase shift fitting procedure. Then, the wavefront result is further refined in an iterative way, by fitting the sequence of interferograms to the physical model of the interferometer cavity with the linear regression technique. Finally, the wavefront result related to the actual surface profile is retrieved after removing the aberrations due to the surface misalignment and the imaging distortion. Both simulations and experiments with the ZYGO interferometer have been carried out to validate the proposed method, with experimental accuracies better than 0.004λ RMS achieved. The proposed method provides a feasible way to spherical surface testing without the use of any phase-shifting devices, while retaining good accuracy and robust convergence performance.