The present follow-up study aims at assessing the longitudinal changes in muscle quality after an interval of 9.45 years in middle-aged men. In addition, the relative contribution of muscle mass, muscle strength, and muscle power at middle age to these changes was investigated. The results showed a small, though unexpected, increase in total body and leg muscle mass (respectively 0.22 ± 0.04 and 0.29 ± 0.06 % yearly, p < 0.0001), whereas basic strength (-0.71 to -0.87 % yearly, p < 0.0001) and velocity-dependent strength and power (-1.19 to -1.86 % yearly, p < 0.0001) declined. Consequently, muscle quality, defined as the ratio of basic strength or velocity-dependent strength and power to muscle mass decreased (-1.46 to -2.43 % yearly, p < 0.0001) from baseline to follow-up. We found that baseline basic strength is a strong determinant of the decline in muscle quality basic strength with advancing age, whereas only a small part of the age-associated decline in muscle quality based on velocity-dependent strength and power could be explained. To conclude, our results indicate that muscle becomes less efficient at middle age and that baseline muscle strength is a strong predictor of this change. These findings imply that unmeasured neural factors, influencing both contraction speed and the capacity of muscle to produce strength, are possibly other involved determinants. Therefore, timely interventions including strength training and higher-velocity strength training at middle age are recommended.