The interaction between molecular (atomic) electron(s) and the vacuum field of a reflective cavity generates significant interest, thanks to the rapid developments in nanophotonics. Such interaction which lies within the realm of cavity quantum electrodynamic can substantially affect the transport properties of molecular systems. In this work, we consider a nonadiabatic electron transfer process in the presence of a cavity mode. We present a generalized framework for the interaction between a charged molecular system and a quantized electromagnetic field of a cavity and apply it to the problem of electron transfer between a donor and an acceptor placed in a confined vacuum electromagnetic field. The effective system Hamiltonian corresponds to a unified Rabi and spin-boson model which includes a self-dipole energy term. Two limiting cases are considered: one where the electron is assumed much faster than the cavity mode and another in which the electron tunneling time is significantly larger than the mode period. In both cases, a significant rate enhancement can be produced by coupling to the cavity mode in the Marcus inverted region. The results of this work offer new possibilities for controlling electron transfer processes using visible and infrared plasmonics.