In this work, we present a novel technique to directly measure the phase shift of the optical signal scattered by single plasmonic nanoparticles in a diffraction-limited laser focus. We accomplish this by equipping an inverted confocal microscope with a Michelson interferometer and scanning single nanoparticles through the focal volume while recording for each pixel interferograms of the scattered and a reference wave. For the experiments, lithographically prepared gold nanorods were used, since their plasmon resonances can be controlled via their aspect ratio. We have developed a theoretical model for image formation in confocal scattering microscopy for nanoparticles considerably smaller than the diffraction limited focus. We show that the phase shift observed for particles with different longitudinal particle plasmon resonances can be well explained by the harmonic oscillator model. The direct measurement of the phase shift can further improve the understanding of the elastic scattering of individual gold nanoparticles with respect to their plasmonic properties.