Background: Valid methods for providing detailed formative feedback on direct laryngoscopy and endotracheal intubation (ETI) performance do not exist. We are developing an observation-based assessment tool for measuring performance and providing feedback during ETI.
Methods: Based on the literature and interviews of experts, we proposed an initial ETI metric with 22 items. Six anesthesiology experts used it to assess the quality of ETI performance in videotaped intubations. Following metric revisions, 2 expert groups assessed 2 collections of videos (27 total) using the revised metric. Two reference standards for comparison with metric scores were created with a third and fourth group of experts; (1) an average global rating (1-100) of each ETI performance and (2) average rank-ordered performance from best to worst. Rater agreement and correlations between the 2 methods were calculated. Regression analysis determined items that optimally discriminated quality. When calculating a score based on all clinically important terms, multiple weightings were evaluated.
Results: Metric items had high average rater agreement (80%) with intraclass correlation coefficients averaging 0.83. Correlations of the reference rank and score were high for both video collections (-0.96, P < .05, and -0.95, P < .05). Regression coefficients for different item weighting methods indicated strong relationships with global ratings (averaging r = 0.89, P < .05) and rankings averaging -0.85, P < .05). Prediction of global ratings using regression achieved high accuracy (R 2 = 0.8218).
Conclusions: High observer agreement and strong correlations between metric and rank data support the validity of using this metric to assess ETI performance. Different weighting models yielded scores that correlated strongly with the ratings and ranks from global assessment. When using the metric to predict competency, a 3-item regression model is most accurate in predicting a global score.
Keywords: Regression analysis; endotracheal intubation; training rubric; video analysis.
© 2020 Society for Education in Anesthesia.