Gait speed is associated with multiple adverse outcomes of aging. White matter hyperintensities (WMH) on magnetic resonance imaging (MRI) have been associated with gait speed, though few studies have examined changes in gait speed over time in population-based studies comprising participants from diverse cultural backgrounds. The purpose of this study was to examine the association between a decline in gait speed and total and regional WMH volumes in a community-based study of aging. Participants (n = 701) underwent gait-speed measurement via a 4-m walk test at the time of initial enrollment and MRI at a second time interval (mean 4.7 [SD = 0.5] years apart). Logistic regression was used to examine the association between large WMH volume and regional WMH volume with gait speed <0.5 m/s (abnormal speed), and a transition to abnormal gait speed. Analyses were adjusted for demographic and clinical factors. Large WMH volume was associated with abnormal gait speed and a transition to abnormal gait speed between the two visits, but not after adjustment for modifiable vascular disease risk factors. Increased frontal lobe WMH volume was associated with abnormal gait speed and transition to abnormal gait speed, but not in adjusted models. WMH are associated with slowing of gait over time. Prevention of WMH presents a potential strategy for the prevention of gait speed decline.