The study aims to investigate associations between adverse childhood psychosocial exposures and declarative memory, language, and executive function in adults with secondary schooling or more and without dementia. In 361 participants from the Pró-Saúde Study, we estimated associations between maternal educational attainment, principal source of the family´s income, food insecurity, and childhood family structure and performance in learning, word recall, and semantic and phonemic verbal fluency tests using multiple linear regression models. Individuals whose mother was the family breadwinner (mean difference: -1.97, 95%CI: -3.27; -0.72) and head-of-household (mean difference: -1.62, 95%CI: -2.89; -0.35) or who lived with a non-parental caregiver or in institutions in childhood (mean difference: -2.19, 95%CI: -4.29; -0.09) showed a reduction in the mean number of words in language and memory in adulthood. The results provide further evidence of the effect of adverse exposures in childhood. Without effective interventions, such exposures are likely to have far-reaching impacts on cognition.