We present a study on the optical losses of Fabry-Pérot cavities subject to realistic transverse mirror misalignment. We consider mirrors of the two most prevalent surface forms: idealised spherical depressions, and Gaussian profiles generated by laser ablation. We first describe the mode mixing phenomena seen in the spherical mirror case and compare to the frequently-used clipping model, observing close agreement in the predicted diffraction loss, but with the addition of protective mode mixing at transverse degeneracies. We then discuss the Gaussian mirror case, detailing how the varying surface curvature across the mirror leads to complex variations in round trip loss and mode profile. In light of the severe mode distortion and strongly elevated loss predicted for many cavity lengths and transverse alignments when using Gaussian mirrors, we suggest that the consequences of mirror surface profile are carefully considered when designing cavity experiments.