Modern society relies on high charge mobility for efficient energy production and fast information technologies. The power factor of a material-the combination of electrical conductivity and Seebeck coefficient-measures its ability to extract electrical power from temperature differences. Recent advancements in thermoelectric materials have achieved enhanced Seebeck coefficient by manipulating the electronic band structure. However, this approach generally applies at relatively low conductivities, preventing the realization of exceptionally high-power factors. In contrast, half-Heusler semiconductors have been shown to break through that barrier in a way that could not be explained. Here, we show that symmetry-protected orbital interactions can steer electron-acoustic phonon interactions towards high mobility. This high-mobility regime enables large power factors in half-Heuslers, well above the maximum measured values. We anticipate that our understanding will spark new routes to search for better thermoelectric materials, and to discover high electron mobility semiconductors for electronic and photonic applications.