For results from large-scale surveys to inform policy and practice appropriately, all participants must interpret and respond to items similarly. While organizers of surveys assessing student outcomes often ensure this for achievement measures, doing so for psychological questionnaires is also critical. We demonstrate this by examining the dimensionality of reading self-concept-a crucial psychological construct for several outcomes-across reading achievement levels. We use Programme for International Student Assessment 2018 data (N = 529,966) and local structural equation models (LSEMs) to do so. Results reveal that reading self-concept dimensions (assessed through reading competence and difficulty) vary across reading achievement levels. Students with low reading achievement show differentiated responses to the two item sets (high competence-high difficulty). In contrast, students with high reading achievement have reconciled responses (high competence-low difficulty). Our results highlight the value of LSEMs in examining factor structure generalizability of constructs in large-scale surveys and call for greater cognitive testing during item development.
Keywords: PISA; factor structure; large-scale survey; local structural equation models; reading self-concept.