It has been long recognized that the single-number quantities presented in the standard ISO 717-2 [(2013) International Organization for Standardization] do not correlate especially well with the subjective judgment of living impact sound sources directed to the floors. The aim of this study was to find single-number quantities which are well associated with the subjective annoyance caused by different impact sounds. Experimental data of laboratory measurements of impact sound insulation of floors and a psychoacoustic experiment was used [Kylliäinen et al. (2017). Acta Acust. Acust. 103, 236-251]. The five studied impact sound types were walking with hard shoes, socks, and soft shoes, super ball bouncing, and chair moving. A fundamental requirement was that the single-number quantities can be expressed as the sum of L'n,w or L'n T, w and a spectrum adaptation term. Reference spectra were derived by the means of a mathematical optimization method. Reference spectra for each sound type were defined separately. An optimized reference spectrum based on all five sound types explained the annoyance of these sound types reasonably well (r2 = 0.93) and better than any of the standardized single number quantities (e.g., r2 = 0.86 for L'n,w + CI,50-2500).