Height loss starting in middle age is reported to be associated with increased all-cause and cardiovascular mortality later in life. However, the mechanisms underlying this association are unclear. Hypoxia and oxidative stress, which are known causes of cardiovascular disease, could be reduced by hemoglobin. Therefore, hemoglobin could be inversely associated with height loss. However, high body mass index (BMI) is a known risk factor for intervertebral disc disorder, a known cause of height loss in adults. High BMI might confound the association between hemoglobin and height loss. Therefore, we performed analyses stratified by BMI status. To clarify the association between hemoglobin and height loss, we conducted a retrospective study of Japanese workers (6,471 men and 3,180 women) aged 40-74 years. Height loss was defined as being in the highest quintile of height decrease per year. In men overall and men with BMI <25 kg/m2, hemoglobin was significantly inversely associated with height loss; but no association was observed for men with high BMI (BMI ≥25 kg/m2) and for women. For men, after adjusting for known cardiovascular risk factors, adjusted odds ratios (ORs) and 95% confidence intervals (CIs) for height loss with each 1 standard deviation (SD) increase in hemoglobin (1.0 g/dL for men and 0.8g/dL for women) were 0.89 (0.83, 0.95) for men overall, 0.82 (0.75, 0.89) for men who do not have high BMI, and 1.01 (0.92, 1.12) for men with high BMI. For women, the corresponding values were 0.97 (0.89, 1.06), 0.98 (0.89, 1.09), and 0.93 (0.75, 1.15) respectively. Hemoglobin is significantly inversely associated with height loss in men who do not have high BMI, but not in men with high BMI or women. These results help clarify the mechanisms underlying height loss, which has been reported to be associated with a higher risk of mortality in adults.