Objective: The goal of this study was to develop an inventory of simulated scenarios that mimic pediatric crises and determine if the resident scores could be used to establish the reliability and validity of a multiple-scenario assessment. The long-term objective is to provide pediatric residents with experiences in the recognition, diagnosis, and management of a range of simulated acute conditions.
Methods: Twenty scenarios were constructed. Each resident participated in 10 scenarios that were scored by 2 independent raters using an action-item checklist and a global score. Debriefing occurred after each scenario. Several analyses were performed to investigate the psychometric adequacy of the scores.
Results: Twenty-nine residents participated. The residents' scores in both sets of 10 scenarios were reliable when using either the checklist or global scoring method (>0.79). Generalizability analyses indicated that the major source of variance in scores was attributable to the scenario and scenario-resident interaction. The variance attributable to the rater was low, yielding high interrater reliability values. The more-experienced residents who participated in the study outperformed the less-experienced interns.
Conclusions: An inventory of critical events was designed to assess pediatric residents' diagnostic and management skills. A reliable measure of ability could be obtained, provided the residents managed multiple scenarios. The residents outscored the interns, providing evidence to support the construct validity of the scores. Additional validity evidence is needed, including studies to determine if this type of training improves physicians' management of real-life critical events.