The decrease of laser-induced damage threshold (LIDT) of optical materials when irradiated with multiple laser pulses is an important phenomenon commonly known as the optical fatigue effect. In case of pulsed femtosecond irradiation fatigue is usually attributed to incubation of laser-induced lattice defects. In this study, standard S-on-1 LIDT test was complimented with in situ time-resolved digital holographic microscopy (TRDHM) to quantitatively investigate fatigue of catastrophic damage for HfO2 and ZrO2 single layer ion-beam-sputtered optical coatings. It was identified that ablation (critical damage) was preceded by exponential increase in optical path length visible as positive phase shift (subcritical damage). Atomic force microscopy was used to show that physical damage originates as localized 100 nm wide nanogrooves perpendicular to laser polarization. A novel link was established between LIDT fatigue and mechanical fatigue crack growth from cyclic loads which allowed construction of a unified numerical fatigue model that reproduced both S-on-1 and TRDHM experimental data.