To improve the performance of binary diffuser-based coherent modulation imaging (CMI), a double-channel optical alignment was proposed. Two diffraction patterns formed by the reflection and transmission of a binary diffuser were simultaneously captured and adopted for iterative reconstruction in combination. The information involved in reflected light, not considered in the traditional single-channel optical alignment, was also reconstructed in this dual-channel binary diffuser-based coherent modulation imaging (DB-CMI). The reconstruction quality and speed were improved and verified by both numerical simulations and proof-of-principle experiments. Therefore, DB-CMI improves traditional CMI and provides a powerful tool for quantitative phase imaging.