Healthy cognitive ageing is a societal and public health priority. Cerebrovascular risk factors increase the likelihood of dementia in older people but their impact on cognitive ageing in younger, healthy brains is less clear. The UK Biobank provides cognition and brain imaging measures in the largest population cohort studied to date. Here we show that cognitive abilities of healthy individuals (N = 22,059) in this sample are detrimentally affected by cerebrovascular risk factors. Structural equation modelling revealed that cerebrovascular risk is associated with reduced cerebral grey matter and white matter integrity within a fronto-parietal brain network underlying executive function. Notably, higher systolic blood pressure was associated with worse executive cognitive function in mid-life (44-69 years), but not in late-life (>70 years). During mid-life this association did not occur in the systolic range of 110-140 mmHg. These findings suggest cerebrovascular risk factors impact on brain structure and cognitive function in healthy people.