We demonstrated a promising route to enhance the performance of inverted organic photovoltaic (OPV) devices by the incorporation of CuGaSe2 (CGS) quantum dots (QDs) into the ZnO buffer layer of P3HT:PCBM-based devices. The modification of QDs provides better band alignment between the organic/cathode interface, improves ZnO crystal quality, and increases photon absorption, leading to more effective carrier transport/collection. By employing this energy-harvesting scheme, short-circuit current density, open-circuit voltage, and fill factor of the OPV device after CGS QD modification are improved by 9.43%, 7.02% and 6.31%, respectively, giving rise to a 23.8% enhancement in the power conversion efficiency.