In bulk heterojunction (BHJ) organic solar cells (OSCs) both the electron affinity (EA) and ionization energy (IE) offsets at the donor-acceptor interface should equally control exciton dissociation. Here, we demonstrate that in low-bandgap non-fullerene acceptor (NFA) BHJs ultrafast donor-to-acceptor energy transfer precedes hole transfer from the acceptor to the donor and thus renders the EA offset virtually unimportant. Moreover, sizeable bulk IE offsets of about 0.5 eV are needed for efficient charge transfer and high internal quantum efficiencies, since energy level bending at the donor-NFA interface caused by the acceptors' quadrupole moments prevents efficient exciton-to-charge-transfer state conversion at low IE offsets. The same bending, however, is the origin of the barrier-less charge transfer state to free charge conversion. Our results provide a comprehensive picture of the photophysics of NFA-based blends, and show that sizeable bulk IE offsets are essential to design efficient BHJ OSCs based on low-bandgap NFAs.