In this paper, we explain in detail the wavelength dependence of the elastic scattering pattern of individual, optically isolated gold nanorods by using confocal microscopy in combination with higher order laser modes, i.e., radially/azimuthally polarized laser modes. We demonstrate that the spectral dependence of the scattering pattern is mostly caused by the relative strength of the gold nanorods' plasmonic modes at different wavelengths. Since the gold nanorods' plasmonic modes are determined by the particles' geometrical parameter, e.g., size and aspect ratio, as well as the refractive index of the surrounding medium, we show that the spectral dependence of the scattering pattern is a simple, not invasive way to determine, e.g., the gold nanorod aspect ratio or physical variation of the local environment. Thus, a further development of spectral imaging of gold nanorods can lead to the employment of this technique in biomedical assays involving also living samples.