Androgen replacement therapy (ART) efficacy on late-onset hypogonadism (LOH) has been widely investigated in Western countries; however, it remains controversial whether ART can improve health and prolong active lifestyles. We prospectively assessed long-term ART effects on the physical and mental statuses of aging men with LOH in Japan. The primary endpoint was health-related quality of life assessed by questionnaires. Secondary endpoints included glycemic control, lipid parameters, blood pressure, waist circumference, body composition, muscular strength, International Prostate Symptom Scores (IPSS), International Index of Erectile Function-5 (IIEF-5) scores, and serum prostate-specific antigen levels. Of the 1637 eligible volunteers, 334 patients > 40 years with LOH were randomly assigned to either the ART (n = 169) or control groups (n = 165). Fifty-two weeks after the initial treatment, ART significantly affected the role physical subdomain of the short form-36 health survey (SF-36) scale (P = 0.0318). ART was also associated with significant decreases in waist circumstance (P = 0.002) and serum triglyceride (TG) (P = 0.013) and with significant increases in whole-body and leg muscle mass volumes (P = 0.071 and 0.0108, respectively), serum hemoglobin (P < 0.001), IPSS voiding subscore (P = 0.0418), and the second question on IIEF-5 (P = 0.0049). There was no significant difference between the groups in terms of severe adverse events. In conclusion, in patients with LOH, long-term ART exerted beneficial effects on Role Physical subdomain of the SF-36 scale, serum TG, waist circumstance, muscle mass volume, voiding subscore of IPSS, and the second question of IIEF-5. We hope our study will contribute to the future development of this area.