The interaction of light with metal nanoparticles leads to novel phenomena mediated by surface plasmon excitations. In this article we use single molecules to characterize the interaction of surface plasmons with light, and show that such interaction can strongly modulate the polarization of the emitted light. The simplest nanostructures that enable such polarization modulation are asymmetric silver nanocrystal trimers, where individual Raman scattering molecules are located in the gap between two of the nanoparticles. The third particle breaks the dipolar symmetry of the two-particle junction, generating a wavelength-dependent polarization pattern. Indeed, the scattered light becomes elliptically polarized and its intensity pattern is rotated in the presence of the third particle. We use a combination of spectroscopic observations on single molecules, scanning electron microscope imaging, and generalized Mie theory calculations to provide a full picture of the effect of particles on the polarization of the emitted light. Furthermore, our theoretical analysis allows us to show that the observed phenomenon is very sensitive to the size of the trimer particles and their relative position, suggesting future means for precise control of light polarization on the nanoscale.