We study surface plasmon polariton lifetimes in two-dimensional arrays of blind holes on gold surfaces. The lifetimes are determined from the line widths of the resonant dips appearing in the specular reflection spectra. We find a strong dependence of lifetimes on the resonant wavelength and the hole geometry. Through both experiments and numerical simulations, we analyze the validity of the Rayleigh approximation and also explore the range of wavelengths and geometric parameters where it fails. Finally, we show that, within the range of geometries and wavelengths considered in our experiments, the behavior of surface plasmon polariton lifetimes can be understood as resulting from the interplay between the intrinsic metal absorption and the scattering of surface waves by single isolated holes.