Correlation of Narrative Evaluations to Clerkship Grades Using Statistical Sentiment Analysis

Douglas R Rice; Gregory N Rice; Charles Baugh; Robert L Cloutier

doi:10.1007/s40670-022-01654-2

Correlation of Narrative Evaluations to Clerkship Grades Using Statistical Sentiment Analysis

Med Sci Educ. 2022 Oct 12;32(6):1397-1403. doi: 10.1007/s40670-022-01654-2. eCollection 2022 Dec.

Authors

Douglas R Rice¹, Gregory N Rice², Charles Baugh³, Robert L Cloutier⁴

Affiliations

¹ Department of Medicine, Oregon Health & Science University, 3181 Sam Jackson Park Road, Portland, OR 97239 USA.
² Department of Statistics and Actuarial Science, University of Waterloo, 200 University Avenue West, Waterloo, ON N2L 3G1 Canada.
³ Department of Family Medicine, Adventist Health 260 Hospital Drive, Suite 103, Ukiah, CA 95482 USA.
⁴ Department of Emergency Medicine and Pediatrics, Oregon Health & Science University, 3181 Sam Jackson Park Road, Portland, OR 97239 USA.

Abstract

Introduction: Narrative evaluations are essential components of medical student assessment. This study evaluated how well narrative clerkship evaluation word choice correlated with an assigned letter grade.

Methods: One hundred clerkship evaluations, 50 from family medicine (FM) and 50 from internal medicine (IM), with even distribution of "Honors" and "Near-Honors" among medical students that graduated in 2020 from the Oregon Health and Science University (OHSU) were examined. A textual sentiment analysis, which evaluates positive and negative word choice, was used to determine each evaluation's collective sentiment. An average sentiment score and character count were calculated for Honors and Near-Honors evaluations from both clerkship disciplines. Sentiment word totals were used to form "word clouds" that highlight the most frequent word selections.

Results: While sentiment scores positively correlated with the assigned grade, there was no statistically significant difference between the average sentiment scores among Honors and Near Honors graded evaluations within the FM or IM clerkship evaluation sets. There was no significant difference in evaluation character length among the assigned grades. Among FM evaluations, "outstanding" and "excellent" were the two most common sentiment words used in both Honors and Near-Honors. Among IM evaluations, outstanding and excellent were most commonly used in Honors evaluations, while "excellent" and "good" were most common in Near-Honors.

Conclusion: This study outlines a novel text analysis method for analyzing narrative evaluation association with assigned grade that other institutions can utilize. Sentiment word choices are not significantly different among Honors and Near Honors clerkship narrative evaluations.

Supplementary information: The online version contains supplementary material available at 10.1007/s40670-022-01654-2.

Keywords: Clinical education; Evaluation; Feedback; Grades; Sentiment.

© The Author(s) under exclusive licence to International Association of Medical Science Educators 2022, Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.