The discovery of magnetic fields close to the M87 black hole using very long baseline interferometry by the Event Horizon Telescope collaboration utilized the novel concept of "closure traces," that are immune to element-based aberrations. We take a fundamentally new approach to this promising tool of polarimetric very long baseline interferometry, using ideas from the geometric phase and gauge theories. The multiplicative distortion of polarized signals at the individual elements are represented as gauge transformations by general 2×2 complex matrices, so the closure traces now appear as gauge-invariant quantities. We apply this formalism to polarimetric interferometry and generalize it to any number of interferometer elements. Our approach goes beyond existing studies in the following respects: (1) we use triangular combinations of correlations as basic building blocks of invariants, (2) we use well-known symmetry properties of the Lorentz group to transparently identify a complete and independent set of invariants, and (3) we do not need autocorrelations, which are susceptible to large systematic biases, and therefore unreliable. This set contains all the information, immune to corruption, available in the interferometer measurements, thus providing important robust constraints for interferometric studies.