Adjuvant chemotherapy (CT) for breast cancer (BC) is associated with very late side-effects on brain function and structure. However, little is known about neurotoxicity of specific treatment regimens. To compare neurotoxicity profiles after different treatment strategies, we used neurocognitive testing and multimodality MRI in BC survivors randomized to high-dose (HI), conventional-dose (CON-) CT or radiotherapy (RT) only and a healthy control (HC) group. BC survivors who received CON-CT (n = 20) and HC (n = 20) were assessed using a neurocognitive test battery and multimodality MRI including 3D-T1, Diffusion Tensor Imaging (DTI) and 1H-MR spectroscopy (1H-MRS) to measure various aspects of cerebral white (WM) and gray matter (GM). Data were compared to previously assessed groups of BC survivors who received HI-CT (n = 17) and RT-only (n = 15). Testing took place on average 11.5 years post-CT. 3D-T1 showed focal GM volume reductions both for HI-CT and CON-CT compared to RT-only (p < .004). DTI-derived mean diffusivity and 1H-MRS derived N-acetyl aspartate showed WM injury specific to HI-CT but not CON-CT (p < .05). Residual effects were revealed in the RT-only group compared to HC on MRI and neurocognitive measurements (p < .05). Ten years after adjuvant CT for BC lower cerebral GM volume was found in HI as well as CON-CT BC survivors whereas injury to WM is restricted to HI-CT. This might indicate that WM brain changes after BC treatment may show more pronounced (partial) recovery than GM. Furthermore, our results suggest residual neurotoxicity in the RT-only group, which warrants further investigation.