Item Selection With Collaborative Filtering in On-The-Fly Multistage Adaptive Testing

Jiaying Xiao; Okan Bulut

doi:10.1177/01466216221124089

Item Selection With Collaborative Filtering in On-The-Fly Multistage Adaptive Testing

Appl Psychol Meas. 2022 Nov;46(8):690-704. doi: 10.1177/01466216221124089. Epub 2022 Aug 28.

Authors

Jiaying Xiao¹, Okan Bulut²

Affiliations

¹ College of Education, University of Washington, Seattle, WA, USA.
² Centre for Research in Applied Measurement and Evaluation, University of Alberta, Edmonton, AB, Canada.

Abstract

An important design feature in the implementation of both computerized adaptive testing and multistage adaptive testing is the use of an appropriate method for item selection. The item selection method is expected to select the most optimal items depending on the examinees' ability level while considering other design features (e.g., item exposure and item bank utilization). This study introduced collaborative filtering (CF) as a new method for item selection in the on-the-fly assembled multistage adaptive testing framework. The user-based CF (UBCF) and item-based CF (IBCF) methods were compared to the maximum Fisher information method based on the accuracy of ability estimation, item exposure rates, and item bank utilization under different test conditions (e.g., item bank size, test length, and the sparseness of training data). The simulation results indicated that the UBCF method outperformed the traditional item selection methods regarding measurement accuracy. Also, the IBCF method showed the most superior performance in terms of item bank utilization. Limitations of the current study and the directions for future research are discussed.

Keywords: collaborative filtering; item selection; measurement accuracy; multistage adaptive testing.