We consider a theory for a two-dimensional interacting conduction electron system with strong spin-orbit coupling on the interface between a topological insulator and the magnetic (ferromagnetic or antiferromagnetic) layer. For the ferromagnetic case we derive the Landau-Lifshitz equation, which features a contribution proportional to a fluctuation-induced electric field obtained by computing the topological (Chern-Simons) contribution from the vacuum polarization. We also show that fermionic quantum fluctuations reduce the critical temperature T[over ˜](c) at the interface relative to the critical temperature T(c) of the bulk, so that in the interval T[over ˜](c)≤T<T(c) it is possible to have a coexistence of gapless Dirac fermions at the interface with a ferromagnetically ordered layer. For the case of an antiferromagnetic layer on a topological insulator substrate, we show that a second-order quantum phase transition occurs at the interface, and compute the corresponding critical exponents. In particular, we show that the electrons at the interface acquire an anomalous dimension at criticality. The critical behavior of the Néel order parameter is anisotropic and features large anomalous dimensions for both the longitudinal and transversal fluctuations.