Perovskite oxides are already widely used in industry and have huge potential for novel device applications thanks to the rich physical behaviour displayed in these materials. The key to the functional electronic properties exhibited by perovskites is often the so-called Jahn-Teller distortion. For applications, an electrical control of the Jahn-Teller distortions, which is so far out of reach, would therefore be highly desirable. Based on universal symmetry arguments, we determine new lattice mode couplings that can provide exactly this paradigm, and exemplify the effect from first-principles calculations. The proposed mechanism is completely general, however for illustrative purposes, we demonstrate the concept on vanadium based perovskites where we reveal an unprecedented orbital ordering and Jahn-Teller induced ferroelectricity. Thanks to the intimate coupling between Jahn-Teller distortions and electronic degrees of freedom, the electric field control of Jahn-Teller distortions is of general relevance and may find broad interest in various functional devices.