We study a way to take into account the scene illumination spectrum during end-to-end optimization of optical-digital hybrid systems that include annular binary phase masks to enhance their depth of field (DoF). We show that a phase mask specifically optimized for wide spectrum panchromatic imaging performs better under this type of illumination than phase masks optimized under monochromatic illumination assumption. Indeed, thanks to spectral averaging, the modulation transfer functions of such a phase mask are very close to each other. This guarantees a very homogeneous image quality across the DoF range, which we demonstrate theoretically and experimentally using a dedicated optical setup.