Fresnel phase matching is a convenient and universal way to phase match nonlinear three-wave mixing by total internal reflection in isotropic materials like common semiconductors. This technique makes use of the large relative phase lag between the interacting waves at total internal reflection, and was suggested by the nonlinear optics pioneers in the 70's; it has been worked out by several teams since then but, quite unexpectedly, has never succeeded in producing enough parametric gain to achieve optical parametric oscillation. We show that this failure stems mostly from a basic law of nonlinear reflection, which leads to a spatial walk-off between the pump and the generated parametric waves, resulting in unexpected destructive interference patterns between the waves while bouncing back and forth between the interfaces. Ray tracing or plane wave analysis gives an incomplete representation of the phenomenon while highly multimodal nonlinear guided wave theory reconciles the different views. Very good agreement between the presented theory and experiments is demonstrated in gallium arsenide samples.