A fundamental understanding of hot-carrier dynamics in photo-excited metal nanostructures is needed to unlock their potential for photodetection and photocatalysis. Despite numerous studies on the ultrafast dynamics of hot electrons, so far, the temporal evolution of hot holes in metal-semiconductor heterostructures remains unknown. Here, we report ultrafast (t < 200 fs) hot-hole injection from Au nanoparticles into the valence band of p-type GaN. The removal of hot holes from below the Au Fermi level is observed to substantially alter the thermalization dynamics of hot electrons, reducing the peak electronic temperature and the electron-phonon coupling time of the Au nanoparticles. First-principles calculations reveal that hot-hole injection modifies the relaxation dynamics of hot electrons in Au nanoparticles by modulating the electronic structure of the metal on timescales commensurate with electron-electron scattering. These results advance our understanding of hot-hole dynamics in metal-semiconductor heterostructures and offer additional strategies for manipulating the dynamics of hot carriers on ultrafast timescales.