Simulation-based Assessment to Measure Proficiency in Mechanical Ventilation among Residents

Fatima K Hayashi; Mayson L A Sousa; Marcos V F Garcia; Bruno R Macedo; Juliana C Ferreira

doi:10.34197/ats-scholar.2021-0130OC

Simulation-based Assessment to Measure Proficiency in Mechanical Ventilation among Residents

ATS Sch. 2022 Jun 30;3(2):204-219. doi: 10.34197/ats-scholar.2021-0130OC. eCollection 2022 Jun.

Authors

Fatima K Hayashi¹, Mayson L A Sousa^{1

2}, Marcos V F Garcia^{1

3}, Bruno R Macedo¹, Juliana C Ferreira^{1

4}

Affiliations

¹ Divisao de Pneumologia, Instituto do Coracao, Hospital das Clinicas HCFMUSP, Faculdade de Medicina, Universidade de Sao Paulo, Sao Paulo, Brazil.
² Interdepartmental Division of Critical Care Medicine, St. Michael's Hospital, University of Toronto, Toronto, Ontario, Canada.
³ Department of Internal Medicine, Cleveland Clinic Foundation, Fairview Hospital, Cleveland, Ohio; and.
⁴ Intensive Care Unit, AC Camargo Cancer Center, São Paulo, Brazil.

Abstract

Background: Mechanical ventilation (MV) skills are essential for clinicians caring for critically ill patients, yet few training programs use structured curricula and appropriate assessments. Objective structured clinical exams (OSCEs) have been used to assess clinical competency in many areas, but there are no OSCE models focused on MV.

Objective: To develop and validate a simulation-based assessment (SBA) with an OSCE structure to assess baseline MV competence among residents and identify knowledge gaps.

Methods: We developed an SBA using a lung simulator and a mechanical ventilator, and an OSCE structure, with six clinical scenarios in MV. We included internal medicine residents at the beginning of their rotation in the respiratory intensive care unit (ICU) of a university-affiliated hospital. A subset of residents was also evaluated with a validated multiple-choice exam (MCE) at the beginning and at the end of the ICU rotation. Scores on both assessments were normalized to range from 0 to 10. We used Cronbach's α coefficient to assess reliability and Spearman correlation to estimate the correlation between the SBA and the MCE.

Results: We included 80 residents, of whom 42 also completed the MCE examinations. The final version of the SBA had 32 items, and the Cronbach's α coefficient was 0.72 (95% confidence interval [CI], 0.64-0.81). The average SBA score was 6.2 ± 1.3, and performance was variable across items, with 80% correctly adjusting initial ventilatory settings and only 12% correctly identifying asynchrony. The MCE had 24 questions, and the average score was 7.6 ± 2.4 at the beginning of the rotation and 8.2 ± 2.3 at the end of the rotation (increase of 0.6 points; 95% CI, 0.30-0.90; P < 0.001). There was moderate correlation between the SBA and the MCE (rho = 0.41; P = 0.002).

Conclusion: We developed and validated an objective structured assessment on MV using a pulmonary simulator and a mechanical ventilator addressing the main competencies in MV. The performance of residents in the SBA at the beginning of an ICU rotation was lower than the performance in MCE, highlighting the need for greater emphasis on practical skills in MV during residency.

Keywords: competency-based education; educational measurement; mechanical ventilation; medical education; simulation training.