It is shown that the decay of the weakly coupled to radiation (dark) modes of subwavelength plasmonic nanostructures is strongly nonexponential. Their lifetime is overestimated by conventional exponential relaxation time obtained in the standard Markovian approximation. These effects are manifestations of the strong dispersion and near-field feedback. The developed theoretical framework introduces an ensemble of local relaxation degrees of freedom coupled to plasmonic mode in order to describe its decay due to material losses. The macroscopic description of the decay process leads to the specific memory function of the system, evaluated from the modal and material dispersions of the plasmonic nanostructure. Proper knowledge of the relaxation behavior is vital for various applications relying on light-matter interactions of emitters with nanoscale objects, such as fluorescence manipulation, bio-imaging, sensing, spasers, sub-diffraction optics, Raman scattering, and quantum optics.