Systems of interacting charges and fields are ubiquitous in physics. Recently, it has been shown that Hamiltonians derived using different gauges can yield different physical results when matter degrees of freedom are truncated to a few low-lying energy eigenstates. This effect is particularly prominent in the ultra-strong coupling regime. Such ambiguities arise because transformations reshuffle the partition between light and matter degrees of freedom and so level truncation is a gauge dependent approximation. To avoid this gauge ambiguity, we redefine the electromagnetic fields in terms of potentials for which the resulting canonical momenta and Hamiltonian are explicitly unchanged by the gauge choice of this theory. Instead the light/matter partition is assigned by the intuitive choice of separating an electric field between displacement and polarisation contributions. This approach is an attractive choice in typical cavity quantum electrodynamics situations.