We present a quantized pseudomode theory for solving system-level cavity quantum electrodynamics with quantum emitters coupled to plasmonic resonators. Exploiting the Green function response of a silver nanoparticle and an exact solution for the input-output coupling, we study the non-Markovian coupling regimes of multi-mode vacuum Rabi oscillations and coherently pumped multi-photon resonances from the higher lying ladder states of collective plasmon modes. The nonlinear pumping regime also facilitates pronounced population inversion of a single two-level atom through multiphoton resonances dominated by dark plasmons.