We have studied the influence of three different fullerene derivatives on the charge generation and recombination dynamics of polymer/fullerene bulk heterojunction (BHJ) solar cell blends. Charge generation in APFO3/[70]PCBM and APFO3/[60]PCBM is very similar and somewhat slower than charge generation in APFO3/[70]BTPF. This difference qualitatively matches the trend in free energy change of electron transfer estimated from the LUMO energies of the polymer and fullerene derivatives. The first order (geminate) charge recombination rate is significantly different for the three fullerene derivatives studied and increases in the order APFO3/[70]PCBM < APFO3/[60]PCBM < APFO3/[70]BTPF. The variation in electron transfer rate cannot be explained from the LUMO energies of the fullerene derivatives and single-step electron transfer in the Marcus inverted region and simple considerations of expected trends for the reorganization energy and free energy change. Instead we suggest that geminate charge recombination occurs from a state where electrons and holes have separated to different distances in the various materials because of an initially high charge mobility, different for different materials. In a BHJ thin film this charge separation distance is not sufficient to overcome the electrostatic attraction between electrons and holes and geminate recombination occurs on the nanosecond to hundreds of nanoseconds time scale. In a BHJ solar cell, we suggest that the internal electric field in combination with polarization effects and the dynamic nature of polarons are key features to overcome electron-hole interactions to form free extractable charges.