Chiral magnetism is a fascinating quantum phenomena that has been found in low-dimensional magnetic materials. It is not only interesting for understanding the concept of chirality, but also important for potential applications in spintronics. Past studies show that chiral magnets require both a lack of inversion symmetry and spin-orbit coupling to induce the Dzyaloshinskii-Moriya interaction. Here we report that the combination of inversion symmetry breaking and quantum degeneracy of orbital degrees of freedom will provide a new paradigm to achieve chiral orbital magnetism. By means of density matrix renormalization group calculation, we demonstrate that chiral orbital magnetism can be found when considering bosonic atoms loaded in the p band of an optical lattice in the Mott regime. The high tunability of our scheme is also illustrated through simply manipulating the inversion symmetry of the system for cold atom experimental conditions.