The perturbed free induction decay (PFID) observed in ultrafast infrared spectroscopy was used to unveil the rates at which different vibrational modes of the same atomic-scale defect can interact with their environment. The N_{3}VH^{0} defect in diamond provided a model system, allowing a comparison of stretch and bend vibrational modes within different crystal lattice environments. The observed bend mode (first overtone) exhibited dephasing times T_{2}=2.8(1) ps, while the fundamental stretch mode had surprisingly faster dynamics T_{2}<1.7 ps driven by its more direct perturbation of the crystal lattice, with increased phonon coupling. Further, at high defect concentrations the stretch mode's dephasing rate was enhanced. The ability to reliably measure T_{2} via PFID provides vital insights into how vibrational systems interact with their local environment.