Use of an error-focused checklist to identify incompetence in lumbar puncture performances

Irene W Y Ma; Debra Pugh; Briseida Mema; Mary E Brindle; Lara Cooke; Julie N Stromer

doi:10.1111/medu.12809

Use of an error-focused checklist to identify incompetence in lumbar puncture performances

Med Educ. 2015 Oct;49(10):1004-15. doi: 10.1111/medu.12809.

Authors

Irene W Y Ma^{1

2}, Debra Pugh³, Briseida Mema⁴, Mary E Brindle⁵, Lara Cooke⁶, Julie N Stromer²

Affiliations

¹ Department of Medicine, Faculty of Medicine, University of Calgary, Calgary, Alberta, Canada.
² W21C, Faculty of Medicine, University of Calgary, Calgary, Alberta, Canada.
³ Department of Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada.
⁴ Department of Critical Care Medicine, Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada.
⁵ Department of Surgery, Faculty of Medicine, University of Calgary, Calgary, Alberta, Canada.
⁶ Department of Clinical Neurosciences, Faculty of Medicine, University of Calgary, Calgary, Alberta, Canada.

Abstract

Context: Checklists are commonly used in the assessment of procedural competence. However, on most checklists, high scores are often unable to rule out incompetence as the commission of a few serious procedural errors typically results in only a minimal reduction in performance score. We hypothesised that checklists constructed based on procedural errors may be better at identifying incompetence.

Objectives: This study sought to compare the efficacy of an error-focused checklist and a conventionally constructed checklist in identifying procedural incompetence.

Methods: We constructed a 15-item error-focused checklist for lumbar puncture (LP) based on input from 13 experts in four Canadian academic centres, using a modified Delphi approach, over three rounds of survey. Ratings of 18 video-recorded performances of LP on simulators using the error-focused tool were compared with ratings obtained using a published conventional 21-item checklist. Competence/incompetence decisions were based on global assessment. Diagnostic accuracy was estimated using the area under the curve (AUC) in receiver operating characteristic analyses.

Results: The accuracy of the conventional checklist in identifying incompetence was low (AUC 0.11, 95% confidence interval [CI] 0.00-0.28) in comparison with that of the error-focused checklist (AUC 0.85, 95% CI 0.67-1.00). The internal consistency of the error-focused checklist was lower than that of the conventional checklist (α = 0.35 and α = 0.79, respectively). The inter-rater reliability of both tools was high (conventional checklist: intraclass correlation coefficient [ICC] 0.99, 95% CI 0.98-1.00; error-focused checklist: ICC 0.92, 95% CI 0.68-0.98).

Conclusions: Despite higher internal consistency and inter-rater reliability, the conventional checklist was less accurate at identifying procedural incompetence. For assessments in which it is important to identify procedural incompetence, we recommend the use of an error-focused checklist.

Publication types

Research Support, Non-U.S. Gov't

MeSH terms

Checklist*
Clinical Competence*
Humans
Medical Errors*
Observer Variation
Spinal Puncture* / standards
Video Recording