On orthopedic surgical skill prediction--the limited value of traditional testing

Abstract

Objectives: Primary: to assess the utility of our distal radius fracture repair model as a tool for examining residents' surgical skills. Secondary: to compare the residents' ability to achieve specific biomechanically measured fracture stability with traditional test scores.

Design: Our laboratory pioneered a model that measures biomechanical qualities of a repaired distal radius fracture. Before participation, all residents to be tested completed specified knowledge examinations. During the laboratory exercise, proctors observed each resident and completed Objective Structured Assessment of Technical Skills forms. At the completion of the laboratory, each specimen was tested biomechanically. Written examinations were completed in a proctored setting and computer examinations at home following the honor system. The laboratory exercise had adequate space and materials and allowed 60 minutes to complete the procedure. Residents had equal access to x-ray imaging.

Setting: The examination environment of the study resembled an operating room.

Participants: Postgraduate years 3 and 4 orthopedic residents in our program were asked to participate. The institutional review board reviewed and approved the study as exempt.

Results: Fracture repair constructs capable of resisting loads expected during rehabilitation were created by approximately half the residents tested. However, traditional written and computer-based testing methods failed to predict which resident's fracture construct would pass the biomechanical testing. Prior in vivo similar case experience was not predictive.

Conclusions: The idea that "book smart does not equal street smart" applies to the tested model. To measure surgical skill acquisition and increase public safety related to surgery, it will be necessary to employ new and specific examination methods that identify the skill to be acquired and test the acquisition of this skill as precisely as possible.

Keywords: Medical Knowledge; Patient Care; Practice-Based Learning and Improvement; criterion-based skill testing; resident education; standardized; surgical competence.