Intervention and policy approaches targeting the societal factors that affect health literacy (e.g., educational systems) could have promise to improve health outcomes, but little research has investigated these factors. This study examined the associations between self-reported racial composition of prior educational and neighborhood contexts and health literacy among 1,061 English- and Spanish-speaking adult community health center patients. The authors found that self-reported racial composition of high school was a significant predictor of health literacy among those who received schooling in the United States, controlling for race/ethnicity, education, age, country of birth, and survey language. Black and Hispanic patients had significantly lower health literacy than White patients within educational strata among those schooled in the United States. The findings revealed substantial disparities in health literacy. Self-reported racial composition of school context was a significant predictor of health literacy. Transdisciplinary, multilevel intervention approaches are likely to be needed to address the health literacy needs of this population.