Optical waveguide theory is essential to the development of various optical devices. Although there are reports on the theory of optical waveguides with magneto-optical (MO) and magnetoelectric (ME) effects, a comprehensive theoretical analysis of waveguides considering these two effects has not yet been published. In this study, the conventional waveguide theory is extended by considering constitutive relations that account for both MO and ME effects. Using the extended waveguide theory, the propagation properties are also analyzed in a medium where metamaterials and magnetic materials are arranged such that MO and ME effects can be controlled independently. It has been confirmed that the interaction between MO and ME effects occurs depending on the arrangement of certain metamaterials and the direction of magnetization. This suggests a nonreciprocal polarization control that rotates the polarization in only one direction when propagating in plane wave propagation and enhances the nonreciprocal nature of the propagating waves in waveguide propagation.