Recent pump-probe experiments performed on graphene in a perpendicular magnetic field have revealed carrier relaxation times ranging from picoseconds to nanoseconds depending on the quality of the sample. To explain this surprising behavior, we propose a novel symmetry-breaking defect-assisted relaxation channel. This enables scattering of electrons with single out-of-plane phonons, which drastically accelerate the carrier scattering time in low-quality samples. The gained insights provide a strategy for tuning the carrier relaxation time in graphene and related materials by orders of magnitude.