Single-photon emission from the nitrogen-vacancy defect in diamond constitutes one of its many proposed applications. Owing to its doubly degenerate 3E electronic excited state, photons from this defect can be emitted by two optical transitions with perpendicular polarization. Previous measurements have indicated that orbital-selective photoexcitation does not, however, yield photoluminescence with well-defined polarizations, thus hinting at orbital-averaging dynamics even at cryogenic temperatures. Here we employ femtosecond polarization anisotropy spectroscopy to investigate the ultrafast electronic dynamics of the 3E state. We observe subpicosecond electronic dephasing dynamics even at cryogenic temperatures, up to five orders of magnitude faster than dephasing rates suggested by previous frequency- and time-domain measurements. Ab initio molecular dynamics simulations assign the ultrafast depolarization dynamics to nonadiabatic transitions and phonon-induced electronic dephasing between the two components of the 3E state. Our results provide an explanation for the ultrafast orbital averaging that exists even at cryogenic temperatures.