Concern that people who form kidney stones may have reduced bone mineral density (BMD) and increased fracture risk has motivated clinical and population-based studies, but findings are inconsistent. In this cross-sectional study, we use the Third National Health and Nutrition Examination Survey (NHANES III) to determine whether a history of kidney stones (n = 793) is associated with lower femoral neck BMD and whether the association is similar for men and women. We further ask whether dietary calcium modifies the association between kidney stone history and BMD and whether there is an association between kidney stone history and prevalent spine or wrist fracture. We find that men with kidney stone history have lower femoral neck BMD than men without kidney stone history after adjusting for age, body mass index (BMI), race/ethnicity, and other potential confounders. The effect of kidney stone history on BMD is weaker for women. Men with kidney stone history also are more likely to report prevalent wrist and spine fractures. Dietary calcium, represented by usual milk consumption, is associated positively with BMD for both men and women and modifies the effect of kidney stone history on BMD for men. For men who form kidney stones, milk consumption is associated more strongly with femoral neck BMD than for men without such a history. The effect modification is such that the difference in BMD between men with and without kidney stone history is observed only at lower levels of milk consumption.