Three classes of harmonic disorder systems (Lennard-Jones-like glasses, percolators above threshold, and spring disordered lattices) have been numerically investigated in order to clarify the effect of different types of disorder on the mechanism of high frequency sound attenuation. We introduce the concept of frustration in structural glasses as a measure of the internal stress, and find a strong correlation between the degree of frustration and the exponent alpha that characterizes the momentum dependence of the sound attenuation gamma(Q) approximately Qalpha. In particular, alpha decreases from approximately d+1 in low-frustration systems (where d is the spectral dimension) to approximately 2 for high-frustration systems such as the realistic glasses examined.