The aim of this study was to investigate the gender- and body-site-specific factors associated with bone mineral density (BMD) at the femoral neck and lumbar spine in a non-institutionalized population aged ≥50 years characterized by low average calcium intake. The comprehensive data utilized were from the 2010 Fifth Korea National Health and Nutrition Examination Survey, which included health behavior questionnaire, blood and urine tests, dual-energy X-ray absorptiometry, and nutrition intake. The factors associated with BMD at the femoral neck and lumbar spine in both genders were analyzed separately using multiple regression analysis with a stepwise selection. The average daily calcium intake in the male subjects was 565.8 mg and in the female subjects was 443.7 mg. In multiple regression analysis, age, body mass index (BMI), alkaline phosphatase (ALP), lead, daily calcium intake, and cadmium were the significant factors associated with femoral neck BMD in male subjects. BMI, creatinine (Cr), total body fat percentile, lead, ALP, and hypertension were found to be the significant factors associated with lumbar spine BMD in male subjects. In the female subjects, the significant factors associated with femoral neck BMD were age, BMI, ALP, house income, and total body fat percentile, while menopause, Cr, mercury, house income, BMI, and ALP were found to be the significant factors associated with lumbar spine BMD. In conclusion, different factors were associated with BMD depending on gender and the body site tested (femoral neck or lumbar spine). These gender- and body-site-specific factors need to be considered for the prevention and management of osteoporosis.