Spin-lattice relaxation within the nitrogen-vacancy (NV) center's electronic ground-state spin triplet limits its coherence times, and thereby impacts its performance in quantum applications. We report measurements of the relaxation rates on the NV center's |m_{s}=0⟩↔|m_{s}=±1⟩ and |m_{s}=-1⟩↔|m_{s}=+1⟩ transitions as a function of temperature from 9 to 474 K in high-purity samples. We show that the temperature dependencies of the rates are reproduced by an ab initio theory of Raman scattering due to second-order spin-phonon interactions, and we discuss the applicability of the theory to other spin systems. Using a novel analytical model based on these results, we suggest that the high-temperature behavior of NV spin-lattice relaxation is dominated by interactions with two groups of quasilocalized phonons centered at 68.2(17) and 167(12) meV.