Nanoscale surface corrugation strongly determines the plasmonic response of gold nanoparticles with dimensions of several tens of nanometres. Scattering spectra of individual spheres with a rough surface were found to red-shift and broaden. The plasmon modes exhibited quadrupole damping, in contrast to particles with smooth surfaces. Additionally, rougher spheres display a higher SERS activity, which demonstrates the crucial role of nanoscale surface texturing on the plasmonic response of gold particles.