The allostatic load index quantifies the cumulative multisystem physiological response to chronic everyday stress, and includes cardiovascular, metabolic and inflammatory measures. Despite its central role in the stress response, research of the effect of allostatic load on the ageing brain has been limited. We investigated the relation of mid-life allostatic load index and multifactorial predictors of stroke (Framingham stroke risk) and diabetes (metabolic syndrome) with voxelwise structural grey and white matter brain integrity measures in the ageing Whitehall II cohort (N = 349, mean age = 69.6 (SD 5.2) years, N (male) = 281 (80.5%), mean follow-up before scan = 21.4 (SD 0.82) years). Higher levels of all three markers were significantly associated with lower grey matter density. Only higher Framingham stroke risk was significantly associated with lower white matter integrity (low fractional anisotropy and high mean diffusivity). Our findings provide some empirical support for the concept of allostatic load, linking the effect of everyday stress on the body with features of the ageing human brain.