SHARP (SHort Answer, Rationale Provision): A New Item Format to Assess Clinical Reasoning

Christopher R Runyon; Miguel A Paniagua; Francine A Rosenthal; Andrea L Veneziano; Lauren McNaughton; Constance T Murray; Polina Harik

doi:10.1097/ACM.0000000000005769

SHARP (SHort Answer, Rationale Provision): A New Item Format to Assess Clinical Reasoning

Acad Med. 2024 May 15. doi: 10.1097/ACM.0000000000005769. Online ahead of print.

Authors

Christopher R Runyon¹, Miguel A Paniagua², Francine A Rosenthal³, Andrea L Veneziano⁴, Lauren McNaughton⁵, Constance T Murray⁶, Polina Harik⁷

Affiliations

¹ C.R. Runyon is senior measurement scientist, Growth and Innovation, National Board of Medical Examiners, Philadelphia, Pennsylvania; ORCID: https://orcid.org/0000-0003-1207-7390.
² M.A. Paniagua was a medical adviser, Test Development, National Board of Medical Examiners, Philadelphia, Pennsylvania, at the time of writing and is now vice president, Medical Education, American College of Physicians, and adjunct professor of medicine, Perelman School of Medicine, The University of Pennsylvania, Philadelphia, Pennsylvania.
³ F.A. Rosenthal is senior editor, Test Development, National Board of Medical Examiners, Philadelphia, Pennsylvania.
⁴ A.L. Veneziano is senior test development analyst, Test Development, National Board of Medical Examiners, Philadelphia, Pennsylvania.
⁵ L. McNaughton is senior test development analyst, Test Development, National Board of Medical Examiners, Philadelphia, Pennsylvania.
⁶ C.T. Murray was director, Test Materials Development, Computer-Based Case Simulations, National Board of Medical Examiners, Philadelphia, Pennsylvania, at the time of this writing and is now retired.
⁷ P. Harik is director of research capabilities, Growth and Innovation, National Board of Medical Examiners, Philadelphia, Pennsylvania.

PMID: 38753971
DOI: 10.1097/ACM.0000000000005769

Abstract

Problem: Many non-workplace-based assessments do not provide good evidence of a learner's problem representation or ability to provide a rationale for a clinical decision they have made. Exceptions include assessment formats that require resource-intensive administration and scoring. This article reports on research efforts toward building a scalable non-workplace-based assessment format that was specifically developed to capture evidence of a learner's ability to provide a justification for a clinical decision that they had made.

Approach: The authors developed a 2-step item format called SHARP (SHort Answer, Rationale Provision), referring to the 2 tasks that comprise the item. In collaboration with physician-educators, the authors integrated short-answer questions into a patient medical record-based item starting in October 2021 and arrived at an innovative item format in December 2021. In this format, a test-taker interprets patient medical record data to make a clinical decision, types in their response, and pinpoints medical record details that justify their answers. In January 2022, a total of 177 fourth-year medical students, representing 20 U.S. medical schools, completed 35 SHARP items in a proof-of-concept study.

Outcomes: Primary outcomes were item timing, difficulty, reliability, and scoring ease. There was substantial variability in item difficulty, with the average item answered correctly by 44% of students (range, 4%-76%). The estimated reliability (Cronbach α) of the set of SHARP items was 0.76 (95% CI, 0.70-0.80). Item scoring is fully automated, minimizing resource requirements.

Next steps: A larger study is planned to gather additional validity evidence about the item format. This study will allow comparisons between performance on SHARP items and other examinations, the examination of group differences in performance, and possible use cases for formative assessment purposes. Cognitive interviews are also planned to better understand the thought processes of medical students as they work through the SHARP items.