A Comparison of Remote vs In-Person Proctored In-Training Examination Administration for Internal Medicine

Thai Q Ong; Becky Krumm; Margaret Wells; Susan Read; Linda Harris; Andrea Altomare; Miguel Paniagua

doi:10.1097/ACM.0000000000005651

A Comparison of Remote vs In-Person Proctored In-Training Examination Administration for Internal Medicine

Acad Med. 2024 Jan 29. doi: 10.1097/ACM.0000000000005651. Online ahead of print.

Authors

Thai Q Ong¹, Becky Krumm², Margaret Wells³, Susan Read⁴, Linda Harris⁵, Andrea Altomare⁶, Miguel Paniagua⁷

Affiliations

¹ T.Q. Ong is senior psychometrician in psychometrics and data analysis, National Board of Medical Examiners, Philadelphia, Pennsylvania.
² B. Krumm is director of self-assessment programs, American College of Physicians, Philadelphia, Pennsylvania.
³ M. Wells is vice president of learning assessment, accreditation, and research, American College of Physicians, Philadelphia, Pennsylvania.
⁴ S. Read is director of research and continuing medical education compliance, American College of Physicians, Philadelphia, Pennsylvania.
⁵ L. Harris is research associate in research and continuing medical education compliance, American College of Physicians, Philadelphia, Pennsylvania.
⁶ A. Altomare is data analyst in psychometrics and data analysis, National Board of Medical Examiners, Philadelphia, Pennsylvania.
⁷ M. Paniagua is vice president of medical education, American College of Physicians, and adjunct professor of medicine, Perelman School of Medicine, The University of Pennsylvania, Philadelphia, Pennsylvania.

PMID: 38277440
DOI: 10.1097/ACM.0000000000005651

Abstract

Purpose: In response to COVID-19, the American College of Physicians provided residents the option to complete the 2020 Internal Medicine In-Training Examination (IM-ITE) via in-person and remote proctoring. This study evaluated the extent to which scores obtained from both testing modalities were comparable.

Method: Data were analyzed from residents from all U.S.-based Accreditation Council for Graduate Medical Education-accredited IM residency programs and participating Canadian and international programs who completed the IM-ITE in 2020. The final sample contained 27,115 IM residents: 9,205 postgraduate year (PGY) 1, 9,332 PGY-2, and 8,578 PGY-3. Testing modality, gender, PGY, time spent on assessment, and native language were used to predict percent-correct scores in a multilevel regression model. This model included all main effects and all 2-way interactions between testing modality and each resident-level demographic variable, allowing those effects to be controlled for.

Results: Of 27,115 residents studied, 11,354 (42%) tested remotely and 15,761 (58%) in person. Across the parameters of interest (main effect of testing modality and 2-way interactions), the only statistically significant effects were the interaction effects between testing mode (interaction effect: -0.61; 95% confidence interval (CI): -1.01, -0.21) and PGY (interaction effect: -0.54; 95% CI: -0.95, -0.13) (P = .002). Differences between in-person and remote predicted scores were slightly larger for PGY-1 than for PGY-2 and PGY-3 residents (controlling for the other predictors in the model), but the magnitude of these differences across residency training was well under a single percentage point. Because these statistically significant effects were deemed educationally nonsignificant, the study concluded that performance did not substantively differ across in-person and remote examinees.

Conclusions: Residents taking the 2020 IM-ITE performed similarly across in-person and remote proctoring. This study provides evidence of score comparability across the 2 testing modalities and supports continued use of remote proctoring for the IM-ITE.