We proposed a digital-optical co-design that can effectively improve the image quality of refractive-diffractive hybrid imaging systems over a wide ambient temperature range. Diffraction theory was used to establish the degradation model and blind deconvolution image recovery algorithm was used to perform recovery for simulated images. The peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) were used to evaluate the algorithm performance. A cooled athermalization dual-band infrared optical system with a double-layer diffractive optical element (DLDOE) was designed, and the results show an overall improvement for both PSNR and SSIM overall the whole ambient temperature range. This demonstrates the effectiveness of the proposed method for the image quality improvement of hybrid optical systems.