Working Towards Competence: A Novel Application of Borderline Regression to a Task-Specific Checklist for Technical Skills in Novices

Claire A Wilson; Saad Chahine; Jacob Davidson; Sumit Dave; Alp Sener; Andrew Rasmussen; Donald H Saklofske; Peter Zhan Tao Wang

doi:10.1016/j.jsurg.2021.05.004

Working Towards Competence: A Novel Application of Borderline Regression to a Task-Specific Checklist for Technical Skills in Novices

J Surg Educ. 2021 Nov-Dec;78(6):2052-2062. doi: 10.1016/j.jsurg.2021.05.004. Epub 2021 Jun 3.

Authors

Claire A Wilson¹, Saad Chahine², Jacob Davidson³, Sumit Dave⁴, Alp Sener¹, Andrew Rasmussen¹, Donald H Saklofske⁵, Peter Zhan Tao Wang⁶

Affiliations

¹ Department of Surgery, Western University, London, Ontario, Canada.
² Faculty of Education, Queen's University, Kingston, Ontario, Canada.
³ Division of Pediatric Surgery, London Health Sciences Centre, London, Ontario, Canada.
⁴ Department of Surgery and Pediatrics, Western University, London, Ontario, Canada.
⁵ Department of Psychology, Western University, London, Ontario, Canada.
⁶ Department of Surgery, Western University, London, Ontario, Canada. Electronic address: peter.wang@lhsc.on.ca.

PMID: 34092532
DOI: 10.1016/j.jsurg.2021.05.004

Abstract

Objective: Demonstrated competence through frequent assessment is an expected goal for progressive development in competency-based medical education curricula. The Objective Structured Assessment of Technical Skill (OSATS) is considered a valid method of formative assessment, but in few instances have standards been set for determining competence. The present study used borderline regression methods to examine standard setting of performance on a complex technical task with novices assessed using an OSATS checklist.

Methods: This was a single institution prospective single arm experimental design study. Participants were 58 non-medical undergraduate students with no previous surgical experience, who observed a computer-based training module on end-to-side vascular anastomosis. Subsequently, participants were provided two 20-minute training sessions, two weeks apart where they received expert feedback whilst performing the task on a low-fidelity model. After each training session, participants completed the task unaided. Sessions were recorded and assessed using an OSATS checklist retrospectively by experts.

Results: Paired t-test analyses indicate that for both the checklist total score (t(52) = 8.05, p < 0.001) and the global rating score (t(53) = 8.15, p < 0.001), individuals performed significantly better in Phase 2. Borderline regression analyses indicated that in Phase 1 (R² = .60) and Phase 2 (R² = .75), the OSATS checklist could adequately capture variation in performance in novices. Further, the checklist could reliably classify novices at three of the five global rating performance levels. Pass rates determined by regression equations improved from Phase 1 to Phase 2 on all global rating levels.

Conclusions: With the increasing focus on competency-based medical education, it is imperative that training programs have the capacity to accurately assess outcomes and set minimum performance standards. Borderline regression methods can accurately differentiate novice learners of varying performance levels before and after training on a complex technical skill task using an OSATS checklist.

Keywords: Borderline regression; Novices; Objective Structured Assessment of Technical Skills; Standard setting; Surgical skill acquisition.

Publication types

Research Support, Non-U.S. Gov't

MeSH terms

Checklist*
Clinical Competence
Humans
Internship and Residency*
Prospective Studies
Retrospective Studies