Elastic scattering of electron vortex beams on magnetic materials leads to a weak magnetic contrast due to Zeeman interaction of orbital angular momentum of the beam with magnetic fields in the sample. The magnetic signal manifests itself as a redistribution of intensity in diffraction patterns due to a change of sign of the orbital angular momentum of the electron vortex beam. While in the atomic resolution regime the magnetic signal is most likely under the detection limits of present transmission electron microscopes, for electron probes with high orbital angular momenta, and correspondingly larger spatial extent, its detection is predicted to be feasible.