The fractional quantum Hall effect has been predicted to occur in the absence of magnetic fields and at high temperature in lattice systems that have flat bands with a nonzero Chern number. We demonstrate that orbital degrees of freedom in frustrated lattice systems lead to a narrowing of topologically nontrivial bands. This robust effect does not rely on fine-tuned long-range hopping parameters and is directly relevant to a wide class of transition-metal compounds.