Spatial confinement can create relaxation bottlenecks by mismatch between electronic and vibrational frequencies. This hypothesis motivated discovery of multiple excitons, which could greatly enhance the efficiency of quantum dot (QD) solar cells. Surprisingly, recent experiments showed no bottleneck. Our time-domain ab initio study of the electron-phonon dynamics rationalizes the fast relaxation in PbSe and CdSe QDs, which have substantially different electronic properties. Atom fluctuations and surface effects lift degeneracies and create dense distributions of electronic levels at all but the lowest energies, while confinement enhances the electron-phonon coupling. The analysis applies to nanomaterials in general, modifying the fundamental view on the electron-phonon interaction.