The interaction of light and matter in metallic nanosystems is mediated by the collective oscillation of surface electrons, called plasmons. After excitation, plasmons are absorbed by the metal electrons through inter- and intraband transitions, creating a highly non-thermal distribution of electrons. The electron population then decays through electron-electron interactions, creating a hot electron distribution within a few hundred femtoseconds, followed by a further relaxation via electron-phonon scattering on the timescale of a few picoseconds. In the spectral domain, hot plasmonic electrons induce changes to the plasmonic resonance of the nanostructure by modifying the dielectric constant of the metal. Here, we report on the observation of anomalously strong changes to the ultrafast temporal and spectral responses of these excited hot plasmonic electrons in hybrid metal/oxide nanostructures as a result of varying the geometry and composition of the nanostructure and the excitation wavelength. In particular, we show a large ultrafast, pulsewidth-limited contribution to the excited electron decay signal in hybrid nanostructures containing hot spots. The intensity of this contribution correlates with the efficiency of the generation of highly excited surface electrons. Using theoretical models, we attribute this effect to the generation of hot plasmonic electrons from hot spots. We then develop general principles to enhance the generation of energetic electrons through specifically designed plasmonic nanostructures that could be used in applications where hot electron generation is beneficial, such as in solar photocatalysis, photodetectors and nonlinear devices.