Any quantum-confined electronic system coupled to the electromagnetic continuum is subject to radiative decay and renormalization of its energy levels. When coupled to a cavity, these quantities can be strongly modified with respect to their values in vacuum. Generally, this modification can be accurately captured by including only the closest resonant mode of the cavity. In the circuit quantum electrodynamics architecture, it is, however, found that the radiative decay rates are strongly influenced by far off-resonant modes. A multimode calculation accounting for the infinite set of cavity modes leads to divergences unless a cutoff is imposed. It has so far not been identified what the source of divergence is. We show here that unless gauge invariance is respected, any attempt at the calculation of circuit QED quantities is bound to diverge. We then present a theoretical approach to the calculation of a finite spontaneous emission rate and the Lamb shift that is free of cutoff.