We describe analytically and numerically the geometric phase arising from nonlinear frequency conversion and show that such a phase can be made non-reciprocal by momentum-dependent photonic transition. Such non-reciprocity is immune to the shortcomings imposed by dynamic reciprocity in Kerr and Kerr-like devices. We propose a simple and practical implementation, requiring only a single waveguide and one pump, while the geometric phase is controllable by the pump and promises robustness against fabrication errors.