The electron dynamics of gold nanorods were systematically studied by using femtosecond transient absorption experiments. Two different excitation wavelengths (400 and 800 nm) have been used as the excitation sources to selectively excite transverse and longitudinal modes. The transient absorption spectra were found to be strongly dependent on the excitation wavelength and fluence. Laser pulses of 800 nm excite the longitudinal mode directly, which cause an increase in the electronic temperatures and subsequent broadening and bleaching of both the longitudinal and transverse modes. Pulses of 400 nm excite both the transverse and longitudinal modes simultaneously. At low excitation fluences, the energy is distributed into two modes according to their steady state extinction coefficients, under which the transient spectra are similar to those under excitation at 800 nm. However, as the excitation fluence exceeds a threshold, the bleaching of the longitudinal plasmon band saturates and the input energies mainly flow to the transverse mode. As a result, the bleaching of the transverse mode increases rapidly. The electron-phonon dynamics show a strong correlation with the bleaching amplitude. We have tried to explain the results with a consistent picture: the bleaching amplitude and electron-phonon relaxation time are directly related to energy distribution into different modes, which are excitation wavelength and fluence dependent. Our studies help to clarify the seemingly inconsistent results in the previous studies by different research groups.