A limited number of studies have associated adjuvant chemotherapy with structural brain changes. These studies had small sample sizes and were conducted shortly after cessation of chemotherapy. Results of these studies indicate local gray matter volume decrease and an increase in white matter lesions. Up till now, it is unclear if non-CNS chemotherapy is associated with long-term structural brain changes. We compared focal and total brain volume (TBV) of a large set of non-CNS directed chemotherapy-exposed breast cancer survivors, on average 21 years post-treatment, to that of a population-based sample of women without a history of cancer. Structural MRI (1.5T) was performed in 184 chemotherapy-exposed breast cancer patients, mean age 64.0 (SD = 6.5) years, who had been diagnosed with cancer on average 21.1 (SD = 4.4) years before, and 368 age-matched cancer-free reference subjects from a population-based cohort study. Outcome measures were: TBV and total gray and white matter volume, and hippocampal volume. In addition, voxel based morphometry was performed to analyze differences in focal gray matter. The chemotherapy-exposed breast cancer survivors had significantly smaller TBV (-3.5 ml, P = 0.019) and gray matter volume (-2.9 ml, P = 0.003) than the reference subjects. No significant differences were observed in white matter volume, hippocampal volume, or local gray matter volume. This study shows that adjuvant chemotherapy for breast cancer is associated with long-term reductions in TBV and overall gray matter volume in the absence of focal reductions. The observed smaller gray matter volume in chemotherapy-exposed survivors was comparable to the effect of almost 4 years of age on gray matter volume reduction. These volume differences might be associated with the slightly worse cognitive performance that we observed previously in this group of breast cancer survivors.