We study the adiabatic dynamics of degenerate quantum states induced by loop paths in a control parameter space. The latter correspond to noisy trajectories if the system is weakly coupled to environmental modes. On top of conventional dynamic dephasing, we find a universal non-Abelian geometric dephasing (NAGD) contribution and express it in terms of the non-Abelian Berry connection and curvature. We show that NAGD implies either decay or amplification of coherences as compared to the coherences when only dynamic dephasing is present. The full NAGD matrix structure can be probed through interference experiments. We outline such a detection scheme for modified Majorana braiding setups.