The polarization property of high harmonics from gallium selenide is investigated using linearly polarized midinfrared laser pulses. With a high electric field, the perpendicular polarization component of the odd harmonics emerges, which is not present with a low electric field and cannot be explained by the perturbative nonlinear optics. A two-dimensional single-band model is developed to show that the anisotropic curvature of an energy band of solids, which is pronounced in an outer part of the Brillouin zone, induces the generation of the perpendicular odd harmonics. This model is validated by three-dimensional quantum mechanical simulations, which reproduce the orientation dependence of the odd-order harmonics. The quantum mechanical simulations also reveal that the odd- and even-order harmonics are produced predominantly by the intraband current and interband polarization, respectively. These experimental and theoretical demonstrations clearly show a strong link between the band structure of a solid and the polarization property of the odd-order harmonics.