Linearly polarized light can exert an orienting torque on plasmonic nanorods. The torque direction has generally been considered to change when the light wavelength passes through a plasmon longitudinal resonance. Here, we use the Maxwell stress tensor to evaluate this torque in general terms. According to distinct light-matter interaction processes, the total torque is decomposed into scattering and extinction torques. The scattering torque tends to orient plasmonic nanorods parallel to the light polarization, independent of the choice of light wavelength. The direction of the extinction torque is not only closely tied to the excitation of plasmon resonance but also depends on the specific plasmon mode around which the light wavelength is tuned. Our findings show that the conventional wisdom that simply associates the total torque with the plasmon longitudinal resonances needs to be replaced with an understanding based on the different torque components and the details of spectral distribution.