Existing computational holographic displays often suffer from limited reconstruction image quality mainly due to ill-conditioned optics hardware and hologram generation software. In this Letter, we develop an end-to-end hardware-in-the-loop approach toward high-quality hologram generation for holographic displays. Unlike other hologram generation methods using ideal wave propagation, ours can reduce artifacts introduced by both the light propagation model and the hardware setup, in particular non-uniform illumination. Experimental results reveal that, compared with classical computer-generated hologram algorithm counterparts, better quality of holographic images can be delivered without a strict requirement on both the fine assembly of optical components and the good uniformity of laser sources.