Can you tell people's cognitive ability level from their response patterns in questionnaires?

Stefan Schneider; Raymond Hernandez; Doerte U Junghaenel; Haomiao Jin; Pey-Jiuan Lee; Hongxin Gao; Danny Maupin; Bart Orriens; Erik Meijer; Arthur A Stone

doi:10.3758/s13428-024-02388-2

Can you tell people's cognitive ability level from their response patterns in questionnaires?

Behav Res Methods. 2024 Mar 25. doi: 10.3758/s13428-024-02388-2. Online ahead of print.

Authors

Stefan Schneider^{1

2

3}, Raymond Hernandez⁴, Doerte U Junghaenel^{4

5

6}, Haomiao Jin⁷, Pey-Jiuan Lee⁴, Hongxin Gao⁷, Danny Maupin⁷, Bart Orriens⁸, Erik Meijer⁸, Arthur A Stone^{4

5}

Affiliations

¹ Dornsife Center for Self-Report Science, and Center for Economic & Social Research, University of Southern California, 635 Downey Way, Los Angeles, CA, 90089-3332, USA. schneids@usc.edu.
² Department of Psychology, University of Southern California, Los Angeles, CA, USA. schneids@usc.edu.
³ Leonard Davis School of Gerontology, University of Southern California, Los Angeles, CA, USA. schneids@usc.edu.
⁴ Dornsife Center for Self-Report Science, and Center for Economic & Social Research, University of Southern California, 635 Downey Way, Los Angeles, CA, 90089-3332, USA.
⁵ Department of Psychology, University of Southern California, Los Angeles, CA, USA.
⁶ Leonard Davis School of Gerontology, University of Southern California, Los Angeles, CA, USA.
⁷ School of Health Sciences, Faculty of Health and Medical Sciences, University of Surrey, Guildford, UK.
⁸ Center for Economic and Social Research, University of Southern California, Los Angeles, CA, USA.

PMID: 38528247
DOI: 10.3758/s13428-024-02388-2

Abstract

Questionnaires are ever present in survey research. In this study, we examined whether an indirect indicator of general cognitive ability could be developed based on response patterns in questionnaires. We drew on two established phenomena characterizing connections between cognitive ability and people's performance on basic cognitive tasks, and examined whether they apply to questionnaires responses. (1) The worst performance rule (WPR) states that people's worst performance on multiple sequential tasks is more indicative of their cognitive ability than their average or best performance. (2) The task complexity hypothesis (TCH) suggests that relationships between cognitive ability and performance increase with task complexity. We conceptualized items of a questionnaire as a series of cognitively demanding tasks. A graded response model was used to estimate respondents' performance for each item based on the difference between the observed and model-predicted response ("response error" scores). Analyzing data from 102 items (21 questionnaires) collected from a large-scale nationally representative sample of people aged 50+ years, we found robust associations of cognitive ability with a person's largest but not with their smallest response error scores (supporting the WPR), and stronger associations of cognitive ability with response errors for more complex than for less complex questions (supporting the TCH). Results replicated across two independent samples and six assessment waves. A latent variable of response errors estimated for the most complex items correlated .50 with a latent cognitive ability factor, suggesting that response patterns can be utilized to extract a rough indicator of general cognitive ability in survey research.

Keywords: Cognitive ability; Item response theory; Questionnaire responding; Task complexity; Worst performance rule.

Grants and funding

R01AG068190/AG/NIA NIH HHS/United States