In this research, an approach called modulation-based structured-illumination microscopy (MSIM) is proposed to measure the surface and thickness profile of thin film layers. With this method, a sinusoidal fringe pattern generated by digital micro-mirror devices (DMD) is projected on the sample. The modulation estimation of the reflected patterns is implemented for characterizing the surface and thickness profile of the sample. The measurement system is relatively simple and only an ordinary objective is enough to achieve imaging of the sample. In addition, the reflected signals come from the back surface of the film create less disturbance to the front surface compared with white-light interferometry. Consequently, they can be easily distinguished and achieve a successful measurement precisely. Both simulation and experiments are carried out to demonstrate the availability of this MISM method. The results are in excellent agreement with commercial stage profiler and the relative uncertainty is less than 10 nm.