We present a microscopic theory that shows the importance of spin-orbit coupling in perovskite compounds with heavy ions. In BiFeO3 (BFO) the spin-orbit coupling at the bismuth ion sites results in a special kind of magnetic anisotropy that is linear in the applied E field. This interaction can convert the cycloid ground state into a homogeneous antiferromagnet, with a weak ferromagnetic moment whose orientation can be controlled by the E-field direction. Remarkably, the E-field control of magnetism occurs without poling the ferroelectric moment, providing a pathway for reduced energy dissipation in spin-based devices made of insulators.