Early-life adversity is pervasive worldwide and represents a potent risk factor for increased mental health burden across the lifespan. However, there is substantial individual heterogeneity in associations between adversity exposure, neurobiological changes, and mental health problems. Accounting for key features of adversity such as the developmental timing of exposure may clarify associations between adversity, neurodevelopment, and mental health. The present study leverages sparse canonical correlation analysis to characterize modes of covariation between age of adversity exposure and the integrity of white matter tracts throughout the brain in a sample of 107 adults. We find that adversity exposure during middle childhood (ages 5-6 and 8-9 in particular) is consistently linked with alterations in white matter tract integrity, such that tracts supporting sensorimotor functions display higher integrity in relation to adversity exposure while tracts supporting cortico-cortical communication display lower integrity. Further, latent patterns of tract integrity linked with adversity experienced across preschool age and middle childhood (ages 4-9) were associated with trauma-related symptoms in adulthood. Our findings underscore that adversity exposure may differentially affect white matter in a function- and developmental-timing specific manner and suggest that adversity experienced between ages 4-9 may shape the development of global white matter tracts in ways that are relevant for adult mental health.