Previous studies of the association between parity and long-term cognitive changes have primarily focused on women and have shown conflicting results. We investigated this association by analyzing data collected on 303,196 subjects from the UK Biobank. We found that in both females and males, having offspring was associated with a faster response time and fewer mistakes made in the visual memory task. Subjects with two or three children had the largest differences relative to those who were childless, with greater effects observed in men. We further analyzed the association between parity and relative brain age (n = 13,584), a brain image-based biomarker indicating how old one's brain structure appears relative to peers. We found that in both sexes, subjects with two or three offspring had significantly reduced brain age compared to those without offspring, corroborating our cognitive function results. Our findings suggest that lifestyle factors accompanying having offspring, rather than the physical process of pregnancy experienced only by females, contribute to these associations and underscore the importance of studying such factors, particularly in the context of sex.