A wavefront sensing method based on a spatial light modulator (SLM) and an incremental binary random sampling (IBRS) algorithm is proposed. In this method, the recording setup is built just by a transmittance SLM and an image sensor. The tested wavefront incident to the SLM plane can be quantitatively retrieved from the diffraction intensities of the wavefront passed through the SLM displaying a IBRS pattern. Because only two modulation states (opaque and transparent) of the SLM are used, the method does not need to know the concrete modulation function of the SLM in advance. In addition by introducing the concept of the incremental random sampling into wavefront sensing, the adaptability of phase retrieving based on the diffraction intensities is significantly improved. To the best of our knowledge, no previous study has used this concept for the same purpose. Some experimental results are given for demonstrating the feasibility of our method.