Dielectric particles in weakly conducting fluids rotate spontaneously when subject to strong electric fields. Such Quincke rotation near a plane electrode leads to particle translation that enables physical models of active matter. In this Letter, we show that Quincke rollers can also exhibit oscillatory dynamics, whereby particles move back and forth about a fixed location. We explain how oscillations arise for micron-scale particles commensurate with the thickness of a field-induced boundary layer in the nonpolar electrolyte. This work enables the design of colloidal oscillators.