Validity Argument for a Simulation-Based Objective Structured Clinical Examination Scenario for Evaluation of Surgical Skills in Trauma

Catalina Ortiz; Francisca Belmar; Rolando Rebolledo; Javier Vela; Caterina Contreras; Martin Inzunza; Juan Pablo Ramos; Analía Zinco; Adnan Alseidi; Julián Varas; Nicolás Jarufe; Pablo Achurra

doi:10.1016/j.jss.2021.06.069

Validity Argument for a Simulation-Based Objective Structured Clinical Examination Scenario for Evaluation of Surgical Skills in Trauma

J Surg Res. 2021 Dec:268:507-513. doi: 10.1016/j.jss.2021.06.069. Epub 2021 Aug 24.

Affiliations

¹ Experimental Surgery and Simulation Center, Department of Digestive Surgery. Pontificia Universidad Católica de Chile, Santiago, Chile.
² Institute for Biomedical Engineering, Pontificia Universidad Católica de Chile, Santiago, Chile.
³ Trauma Surgery and Emergency Medicine Department, Hospital Dr. Sotero del Río, Santiago, Chile.
⁴ Department of Surgery, University of California San Francisco, San Francisco, California.
⁵ Experimental Surgery and Simulation Center, Department of Digestive Surgery. Pontificia Universidad Católica de Chile, Santiago, Chile. Electronic address: achurrapablo@gmail.com.

PMID: 34450554
DOI: 10.1016/j.jss.2021.06.069

Abstract

Background: Trauma is one of the main causes of death globally, and appropriate surgical care is crucial to impact mortality. However, resident-performed trauma cases have diminished in the last 10 years. Simulation-based tools have proven to be effective to evaluate practical skills in a variety of settings. However, there is a lack of evidence regarding proper validation of trauma surgery models.

Objective: The aim of this study was to evaluate under a contemporary validity framework, an objective structured clinical evaluation (OSCE) scenario for the assessment of basic and advanced surgical skills in trauma and emergency surgery.

Methods: An OSCE-type simulation assessment program was developed incorporating six stations representing basic and advanced surgical skills that are essential in trauma surgery. Each station was designed using ex-vivo animal tissue. The stations included basic knots and sutures, bowel resection and anastomosis, vascular end-to-end anastomosis, lung injury repair, cardiac injury repair, and laparoscopic suturing. Eight postgraduate year 2 (PY-2), eight recently graduated surgeons (RGS), and 3 experts were recruited, and their performance was blindly assessed by experts using the validated general rating scale OSATS (Objective Structured Assessment of Technical Skills) as well as the time taken to complete the procedure.

Results: Significant differences were identified among groups. The average OSATS score was 82 for the PY2 group, 113 for the RGS group, and 147 for the experts (P < 0.01). The average procedural time to complete all the stations was 98 minutes for the PY2 group, 68 minutes for the RGS group, and 35 minutes for the expert surgeons (P < 0.01).

Conclusion: An OSCE scenario designed using ex-vivo tissue met 4 out of 5 criteria of the Messick validity framework: content, relation to other variables, response process and consequences of the test. The results show it is a valid strategy for the evaluation of practical skills in trauma surgery.

Keywords: Resident training; Surgical education; Surgical simulation; Trauma surgery.

Publication types

Research Support, Non-U.S. Gov't

MeSH terms

Animals
Clinical Competence
Computer Simulation
Educational Measurement / methods
Internship and Residency*
Laparoscopy*
Models, Anatomic
Physical Examination