The mechanisms producing strong coupling between electric and magnetic order in multiferroics are not always well understood, since their microscopic origins can be quite different. Hence, gaining a deeper understanding of magnetoelectric coupling in these materials is the key to their rational design. Here, we use ultrafast optical spectroscopy to show that the influence of magnetic ordering on quantum charge fluctuations via the double-exchange mechanism can govern the interplay between electric polarization and magnetism in the charge-ordered multiferroic LuFe2O4.