Nonlinear interactions between light waves can exchange energy, linear momentum, and angular momentum. The direction of energy flow between frequency components is usually determined by the conventional phase-matching condition related to the linear momentum. However, the transfer law of orbital angular momentum (OAM) during frequency conversion remains to be elucidated. Here, we demonstrate experimentally that OAM transfer depends strongly on the phase-matching condition defined by both linear and orbital angular momenta. Under different phase-matching configurations, the second-harmonic wave exhibits variable OAM spectral characteristics such as the presence of just a single value or of odd orders only. Our results pave the way toward unveiling the underlying mechanism of nonlinear conversion of OAM states.