Grain growth of nanocrystalline materials is generally thermally activated, but can also be driven by irradiation at much lower temperature. In nanocrystalline ceria and zirconia, energetic ions deposit their energy to both atomic nuclei and electrons. Our experimental results have shown that irradiation-induced grain growth is dependent on the total energy deposited, where electronic energy loss and elastic collisions between atomic nuclei both contribute to the production of disorder and grain growth. Our atomistic simulations reveal that a high density of disorder near grain boundaries leads to locally rapid grain movement. The additive effect from both electronic excitation and atomic collision cascades on grain growth demonstrated in this work opens up new possibilities for controlling grain sizes to improve functionality of nanocrystalline materials.