Compared with whites, blacks develop hypertension earlier in life, progress from prehypertension to hypertension at an accelerated rate, and exhibit greater hypertension-mediated organ damage (e.g., kidney disease, stroke). In this paper, we tested whether the longitudinal associations between elevated systolic blood pressure and disruption of brain white matter structural integrity differ as a function of race. A community sample of 100 middle-aged adults with prehypertension underwent diffusion imaging to quantify indirect metrics of white matter structural integrity, including fractional anisotropy. Blood pressure and diffusion imaging measurements were collected at baseline and at a 2-year follow-up. Regression analyses showed that higher systolic blood pressure at baseline was associated with a decrease in fractional anisotropy over 2 years in blacks only (β = -0.51 [95% CI = -0.85, -0.16], t = -2.93, p = 0.004, ΔR2 = 0.09). These findings suggest that blacks are more susceptible to the impact of systolic prehypertension on white matter structural integrity.