The anomalous Hall effect is a fundamental transport process in solids arising from the spin-orbit coupling. In a quantum anomalous Hall insulator, spontaneous magnetic moments and spin-orbit coupling combine to give rise to a topologically nontrivial electronic structure, leading to the quantized Hall effect without an external magnetic field. Based on first-principles calculations, we predict that the tetradymite semiconductors Bi2Te3, Bi2Se3, and Sb2Te3 form magnetically ordered insulators when doped with transition metal elements (Cr or Fe), in contrast to conventional dilute magnetic semiconductors where free carriers are necessary to mediate the magnetic coupling. In two-dimensional thin films, this magnetic order gives rise to a topological electronic structure characterized by a finite Chern number, with the Hall conductance quantized in units of e2/h (where e is the charge of an electron and h is Planck's constant).