Interrater agreement between student and teacher assessments of endotracheal intubation skills in a self-directed simulation learning environment

Aida Mankute; Laima Juozapaviciene; Justinas Stucinskas; Zilvinas Dambrauskas; Paulius Dobozinskas; Elizabeth Sinz; David L Rodgers; Evelina Pukenyte; Birute Kumpaitiene; Dinas Vaitkaitis

doi:10.1186/s12909-023-04242-z

Interrater agreement between student and teacher assessments of endotracheal intubation skills in a self-directed simulation learning environment

BMC Med Educ. 2023 Apr 18;23(1):256. doi: 10.1186/s12909-023-04242-z.

Affiliations

¹ Department of Emergency Medicine, Lithuanian University of Health Sciences, Kaunas, Lithuania. Aida.Mankute@lsmuni.lt.
² Department ofAnaesthesiology, Lithuanian University of Health Sciences, Kaunas, Lithuania.
³ Department of Orthopaedics and Traumatology, Lithuanian University of Health Sciences, Kaunas, Lithuania.
⁴ Department of Surgery, Lithuanian University of Health Sciences, Kaunas, Lithuania.
⁵ Department of Disaster Medicine, Lithuanian University of Health Sciences, Kaunas, Lithuania.
⁶ Department of Anesthesiology and Perioperative Medicine, Medical Simulation Center, Penn State Health Milton S. Hershey Medical Center, Hershey, USA.
⁷ Interprofessional Simulation Center, Indiana University, Bloomington, USA.
⁸ Department of Infectious Diseases, Lithuanian University of Health Sciences, Kaunas, Lithuania.
⁹ Department of Cardiothoracic and Vascular Surgery, Lithuanian University of Health Sciences, Kaunas, Lithuania.

Abstract

Background: Practical skill assessment is an important part of the learning process to confirm competencies in acquired medical knowledge.

Objective: This study aimed to compare the assessments of endotracheal intubation skills using the HybridLab® methodology between students and teacher in terms of interobserver reliability.

Methods: Reliability analysis was performed with observational data (data are reported according to STROBE guidelines). The study was conducted in two countries, the Lithuanian University of Health Science (LUHS) and Pennsylvania State University (PSU) in the US, between 1 January and 30 June 2020. A total of 92 students (60 from LUHS and 32 from PSU) were trained in endotracheal intubation using an algorithm-driven hybrid learning method. At the end of the training session, the participants had to complete the evaluation scenario, which was assessed by one of the students and evaluated remotely by a single teacher. The student assessment of the endotracheal intubation procedure was compared with the teacher's assessment using correlation and estimation of the intraclass correlation coefficient.

Results: Overall, the medians of the student and teacher assessments were both 100% (0%). Spearman's correlation coefficient between the student and teacher assessments was 0.879 (p = 0.001). The intraclass correlation coefficient used for interobserver variations between the students and teacher was 0.883 (95% confidence interval from 0.824 to 0.923).

Conclusions: The algorithm-driven hybrid learning method allows students to reliably assess endotracheal intubation skills to a level comparable with that of the teacher's evaluation. This learning method has the potential to be a cost-effective and efficient way to provide high-quality education while also saving human resources.

Keywords: HybridLab; Self-directed learning; Skill assessment; Students; Teacher.

MeSH terms

Educational Measurement
Humans
Intubation, Intratracheal
Learning*
Reproducibility of Results
Students*