The quantum spin Hall (QSH) state of matter is usually considered to be protected by time-reversal (TR) symmetry. We investigate the fate of the QSH effect in the presence of the Rashba spin-orbit coupling and an exchange field, which break both inversion and TR symmetries. It is found that the QSH state characterized by nonzero spin Chern numbers C(±) = ±1 persists when the TR symmetry is broken. A topological phase transition from the TR-symmetry-broken QSH phase to a quantum anomalous Hall phase occurs at a critical exchange field, where the bulk band gap just closes. It is also shown that the transition from the TR-symmetry-broken QSH phase to an ordinary insulator state cannot happen without closing the band gap.