Objective: To determine the interrater reliability and construct validity of 3 separate assessment tools for assessing trainee skills in pediatric airway endoscopy simulation.
Design: An Objective Structured Assessment of Technical Skills (OSATS) was developed in which examinees were asked to name and assemble the airway foreign body instruments and retrieve a foreign body from an infant airway mannequin. Each examinee's performance was assessed in a blinded fashion by 3 pediatric otolaryngology faculty at separate residency programs using 3 assessment tools: (1) objective quantifiable measures list (eg, assists needed, forceps openings, foreign body drops), (2) 15-point OSATS checklist, and (3) Global Rating Index for Technical Skills (GRITS). Setting Otolaryngology residency program. Subjects Examinees (medical students, n = 3; otolaryngology residents, n = 17; pediatric otolaryngology faculty, n = 3) and raters (n = 3). Main Outcome Measures Interrater reliability and construct validity. Results Anonymized split-screen videos simultaneously capturing each examinee's instrument handling and the endoscopic videos were created for all 23 examinees. Nineteen videos were chosen for review by 3 raters. The interrater reliability as measured by the intraclass correlation for objective quantifiable measures ranged from 0.46 to 0.98. The intraclass correlation coefficient was 0.95 for the 15-point OSATS checklist and 0.95 for the GRITS; both showed a high degree of construct validity with scores correlating with previous experience. Conclusion Assessment tools for skills assessments must have high interrater reliability and construct validity. When assessing trainee skills in pediatric airway foreign body scenarios, the 15-point OSATS checklist developed by this group or the GRITS meets these criteria.