Physical activity (PA) and cardiorespiratory fitness (CRF) are associated with better cognitive function in late life, but the neural correlates for these relationships are unclear. To study these correlates, we examined the association of both PA and CRF with measures of white matter (WM) integrity in 88 healthy low-fit adults (age 60-78). Using accelerometry, we objectively measured sedentary behavior, light PA, and moderate to vigorous PA (MV-PA) over a week. We showed that greater MV-PA was related to lower volume of WM lesions. The association between PA and WM microstructural integrity (measured with diffusion tensor imaging) was region-specific: light PA was related to temporal WM, while sedentary behavior was associated with lower integrity in the parahippocampal WM. Our findings highlight that engaging in PA of various intensity in parallel with avoiding sedentariness are important in maintaining WM health in older age, supporting public health recommendations that emphasize the importance of active lifestyle.