Contrasting multistage and computer-based testing: score accuracy and aberrant responding

Georgios Sideridis; Hanan Ghamdi; Omar Zamil

doi:10.3389/fpsyg.2023.1288177

Contrasting multistage and computer-based testing: score accuracy and aberrant responding

Front Psychol. 2023 Dec 5:14:1288177. doi: 10.3389/fpsyg.2023.1288177. eCollection 2023.

Authors

Georgios Sideridis^{1

2}, Hanan Ghamdi³, Omar Zamil³

Affiliations

¹ Boston Children's Hospital, Harvard Medical School, Boston, MA, United States.
² Department of Primary Education, National and Kapodistrian University of Athens, Athens, Greece.
³ Education and Training Evaluation Commission, Riyadh, Saudi Arabia.

Abstract

The goal of the present study was to compare and contrast the efficacy of a multistage testing (MST) design using three paths compared to a traditional computer-based testing (CBT) approach involving items across all ability levels. Participants were n = 627 individuals who were subjected to both a computer-based testing (CBT) instrument and a measure constructed using multistage testing to route individuals of low, middle, and high ability to content that was respective to their ability level. Comparisons between the medium of testing involved person ability accuracy estimates and evaluation of aberrant responding. The results indicated that MST assessments deviated markedly from CBT assessments, especially for low- and high-ability individuals. Test score accuracy was higher overall in MST compared to CBT, although error of measurement was enhanced for high-ability individuals during MST compared to CBT. Evaluating response patterns indicated significant amounts of Guttman-related errors during CBT compared to MST using person-fit aberrant response indicators. It was concluded that MST is associated with significant benefits compared to CBT.

Keywords: aberrant responding; carelessness; guessing; item response theory; multistage testing; person fit statistics.

Grants and funding

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This project was funded by ETEC, Riyadh, Saudi Arabia.