Is There Variability in Scoring of Student Surgical OSCE Performance Based on Examiner Experience and Expertise?

Abstract

Objective: To investigate the influence of clinical experience and content expertise on global assessment scores in a Surgical Objective Structured Clinical Exam (OSCE) for senior medical undergraduate students.

Design: Scripted videos of simulated student performance in an OSCE at two standards (clear pass and borderline) were awarded a global score on each of two rating scales by a range of clinical assessors. Results were analysed by examiner experience and content expertise.

Setting: The study was designed in a large Medical School in Ireland. Examiners were consultant and training grade doctors from three university teaching hospitals.

Participants: 147 assessors participated. Of these, 75 (51%) were surgeons and 25 (17%) had sub-speciality surgical expertise directly relevant to the OSCE station. 41 were consultants.

Results: Responsible academic scoring set the benchmark. By multivariable linear regression analysis, neither clinical experience (consultant status) nor relevant content expertise in surgery was independently predictive of assessor grading for either clear pass or borderline student performance. No educational factor (previous examining experience/training, self-rated confidence in assessment or frame of reference) was significant. Assessor gender (male) was associated with award of a fail grade for borderline performance. Trainees were reliable graders of borderline performance but more lenient than the gold standard for clear pass. We report greater agreement with the gold standard score using the global descriptive scale, with strong agreement for all assessors in the borderline case.

Conclusions: Neither assessor clinical experience nor content expertise is independently predictive of grade awarded in an OSCE. Where non-experts or trainees assess, we find evidence for use of a descriptive global score to maximise agreement with expert gold standard, particularly for borderline performance. These results inform the fair and reliable participation of a range of examiners across subspecialty stations in the surgical OSCE format.

Keywords: Medical Knowledge; OSCE (Objective Structured Clinical Examination); Practice-Based Learning and Improvement; Systems-Based Practice; assessment; examiner bias; global rating.