Coherent modulation imaging (CMI) is an effective lensless diffraction imaging method with fast algorithmic convergence and high robustness to data defects. In the reported algorithms for CMI, one important requirement is that the modulator function need to be known a priori; and an additional step for the modulator characterization is required to be carried out in advance by other methods, such as ptychography, which could be cumbersome in practice. Here, we propose an improved algorithm that allows for the transmission function of a completely unknown modulator to be recovered during the same iterative process of image reconstruction. We have verified the method in both simulations and optical experiments. This improvement would turn CMI into a more practical and standalone technique for broader applications in biology and materials science.