The Relationship between In-Training Examinations and Simulation Assessments in an Emergency Medicine Residency Program

Context: Historically, the assessment of medical resident knowledge and skill has occurred through annual in-training examinations and faculty observation during real patient encounters. To improve patient care and the learning environment, medical educators have begun creating simulation experiences for medical residents to perform procedures without using real patients. However, simulation curricula and accompanying assessment techniques have not been standardized. Using a longitudinal record review, in-training examination scores were compared to annualized simulation assessment scores to see if there is any relationship between the assessment types.

Methods: This project was a retrospective eight-year study from a single residency program. The scores were collected from 102 resident academic records from 2007 to 2015 for the annual American Board of Emergency Medicine in-training examination and the resident's annual simulation assessment. Complete data sets were analyzed to determine if a relationship exists between the assessment methods. Then the averages were compared for only the first three years for all students and for all four years for osteopathic residents as they have an additional fourth year of training.

Results: This study showed a lack of relationship between the two assessment types when reviewing three years of data. When the fourth year data is considered, there was a significant relationship between the assessment types.

Conclusions: The performance scores for both types of assessment provide independent information on the resident progress in training. Therefore, they should both be reviewed and considered to appropriately measure the resident's performance. The significance of the fourth year of training for osteopathic residents requires further study.