A 5-year review of feedback on the Philadelphia-area mock ABS certifying examination

Abstract

Objectives: The American Board of Surgery Certifying Examination (ABSCE) is an oral examination designed to evaluate a resident׳s ability to apply their cognitive knowledge to manage a broad range of clinical problems. In this study, we analyze our 5-year experience with a Philadelphia-wide mock oral examination (PMOE).

Setting: The PMOE is organized by the Metropolitan Philadelphia Chapter of the American College of Surgeons and offered annually to all postgraduate year 4/5 residents from the 8 participating Philadelphia general surgery programs. Each examinee is scheduled for 3 consecutive 30-minute examinations given by 2 examiners per room. Overall performance is graded for each interaction using the ABSCE scoring method. Participants are given their "pass/fail" status, and they receive written examiner feedback.

Design: From 2008 to 2013, deidentified examinee scores from both the PMOE and the ABSCE were reviewed; overall pass/fail status was compared using the chi-square statistic for significance. Examinee feedback from 2009 to 2013 was reviewed by 3 independent raters and characterized as commenting upon cognitive knowledge, clinical management, or communication skills. This categorical data were then correlated with pass/fail status and examined using unpaired t tests for significance.

Results: From 2009 to 2013, 189 residents participated in the PMOE with an overall pass rate of 53%, compared with the ABSCE pass rate of 76% for 113 examinees from the Philadelphia area from 2008 to 2013 (χ(2) = 18.8, p < 0.01). A total of 2273 comments were reviewed and categorized from 2009 to 2013. Examinees who failed the PMOE received significantly more feedback pertaining to cognitive knowledge than examinees who passed the examination (p = 0.04).

Conclusion: The PMOE provides residents an opportunity to receive feedback on their performance on a representation of the ABSCE that may be more rigorous than the actual certifying examination. Deficits in cognitive knowledge are a significant determinant of performance on a city-wide mock oral examination.

Keywords: Interpersonal and Communication Skills; Medical Knowledge; Patient Care; certification; educational measurement; general surgery; specialty boards.