Expanding scope of Kirkpatrick model from training effectiveness review to evidence-informed prioritization management for cricothyroidotomy simulation

Victor Kai-Lam Cheung; Nam-Hung Chia; Sze-Sze So; George Wing-Yiu Ng; Eric Hang-Kwong So

doi:10.1016/j.heliyon.2023.e18268

Expanding scope of Kirkpatrick model from training effectiveness review to evidence-informed prioritization management for cricothyroidotomy simulation

Heliyon. 2023 Jul 25;9(8):e18268. doi: 10.1016/j.heliyon.2023.e18268. eCollection 2023 Aug.

Authors

Victor Kai-Lam Cheung^{1

2}, Nam-Hung Chia^{1

3}, Sze-Sze So¹, George Wing-Yiu Ng^{1

4}, Eric Hang-Kwong So^{1

5}

Affiliations

¹ Multi-Disciplinary Simulation & Skills Centre (MDSSC), Queen Elizabeth Hospital, Hong Kong Special Administrative Region.
² Department of Neuroscience, Psychology and Behaviour, University of Leicester, UK.
³ Department of Surgery, Queen Elizabeth Hospital, Hong Kong Special Administrative Region.
⁴ Intensive Care Unit, Queen Elizabeth Hospital, Hong Kong Special Administrative Region.
⁵ Department of Anaesthesiology & Operating Theatre Services, Queen Elizabeth Hospital, Hong Kong Special Administrative Region.

Abstract

Modified Kirkpatrick model has been adopted to evaluate training effectiveness by 6 categories, including activity accounting (training objectives/success in organization change) at Level-0, reaction (satisfaction) at Level-1, learning (acquisition of surgical airway skills) at Level-2, behavior (post-training change in personal strengths) at Level-3, result (organizational or clinical outcomes) at Level-4, and Return on Investment (ROI) or Expectation (ROE) (monetary and societal values following training and other quality and safety related measures) at Level-5. The purpose of this hospital-based prospective observational study was twofold: i) To evaluate potential impacts on monetary and societal values and successful organization change following implementation of advanced Cricothyroidotomy simulator and standardized curriculum in healthcare simulation training, ii) To inform decisions of resource allocation by reviewing overall values and prioritization strategies for i) general surgeon/emergency physician ii) with seniority >5 years and iii) prior porcine training experience based on findings at Kirkpatrick Level-0, Level-4, and Level-5. Seventy doctors and 10 nurses completed Cricothyroidotomy training and follow-up questionnaires within 2021/22. All training usability scoring measured by Scales of Emergency Surgical Airway Simulator (SESAS-17) achieved over 4 out of 5 (Level-4) with effects in favor of emergency physicians or general surgeons (p < .5), regardless of seniority and prior training experience. Success in organization change (Level-0) and cost-effectiveness (Level-5) were hypothetically established using theoretical framework of Gleicher's formula and Roger's Diffusion of Innovation Theory. Overall training effectiveness, in terms of advantage in usability, cost-benefits and successful organizational changes, provided sound evidence to support continuous investment of new curriculum and innovative simulator and "Surgeon-and-emergency-physician-first" policy when it comes to resources allocation strategies for Cricothyroidotomy training. [ACGME competencies: Practice Based Learning and Improvement, Systems Based Practice.].

Keywords: Evidence-based management; Medical simulation; Modified Kirkpatrick; Organizational change; Surgical Cricothyroidotomy; Training effectiveness.