We design and experimentally demonstrate an optical switch based on the interference of plasmonic modes in whispering gallery mode (WGM) antennas. Simultaneous excitation of even and odd WGM modes, enabled by a small symmetry breaking via non-normal illumination, allows switching the plasmonic near field between opposite sides of the antenna, depending on the excitation wavelength used in a wavelength range of 60 nm centered around 790 nm. This proposed switching mechanism is experimentally demonstrated by combining photoemission electron microscopy (PEEM) with a tunable wavelength femtosecond laser source in the visible and infrared.