Scattering processes have played a crucial role in the development of quantum theory. In the field of optics, scattering phase shifts have been utilized to unveil interesting forms of light-matter interactions. Here we investigate the phase shift experienced by a single photon as it scatters into a surface plasmon polariton and vice versa. This coupling phase shift is of particular relevance for quantum plasmonic experiments. Therefore, we demonstrate that the photon-plasmon interaction at a plasmonic slit can be modeled through a quantum-mechanical tritter, a six-port scattering element. We show that the visibilities of a double-slit and a triple-slit interference patterns are convenient observables to characterize the interaction at a slit and determine the coupling phase. Our accurate and simple model of the interaction, validated by simulations and experiments, has important implications not only for quantum plasmonic interference effects, but is also advantageous to classical applications.