The effect of putative cerebrovascular risk factors on gray matter volume in a community-dwelling, non-demented 60- to 64-year-old cohort was investigated. Cranial T1-weighted MRI scans were obtained in 337 adults and voxel-based morphometric analyses were applied to detect regional gray matter volume differences related to hypertension, diabetes, smoking, and hyperlipidemia in men and women, respectively. Hypertension-related gray matter volume reduction was found in right superior, bilateral medial frontal, left superior temporal and left precentral gyri in men. No regional differences in gray matter related to hypertension were seen in women. Conversely, female but not male smokers had more gray matter volume in right fusiform gyrus and right temporal subgyral gray matter. No differences were observed in gray matter volume in association with diabetes or hyperlipidemia for men or women. Our results suggest that there are different patterns of regional effects in gray matter volume in relation to different cerebrovascular risk factors, and sex differences for the same risk factors.