For the modulation-based structured illumination microscopy system, how to obtain modulation distribution with an image has been a research hotspot. However, the existing frequency-domain single-frame algorithms (mainly including the Fourier transform method, wavelet method, etc.) suffer from different degrees of analytical error due to the loss of high-frequency information. Recently, a modulation-based spatial area phase-shifting method was proposed; it can obtain higher precision by retaining high-frequency information effectively. But for discontinuous (such as step) topography, it would be somewhat smooth. To solve the problem, we propose a high-order spatial phase shift algorithm that realizes robust modulation analysis of a discontinuous surface with a single-frame image. At the same time, this technique proposes a residual optimization strategy, so that it can be applied to the measurement of complex topography, especially discontinuous topography. Simulation and experimental results demonstrate that the proposed method can provide higher-precision measurement.