Performance of ChatGPT on the Peruvian National Licensing Medical Examination: Cross-Sectional Study

Javier A Flores-Cohaila; Abigaíl García-Vicente; Sonia F Vizcarra-Jiménez; Janith P De la Cruz-Galán; Jesús D Gutiérrez-Arratia; Blanca Geraldine Quiroga Torres; Alvaro Taype-Rondan

doi:10.2196/48039

Performance of ChatGPT on the Peruvian National Licensing Medical Examination: Cross-Sectional Study

JMIR Med Educ. 2023 Sep 28:9:e48039. doi: 10.2196/48039.

Authors

Javier A Flores-Cohaila^{1

2}, Abigaíl García-Vicente^{3

4}, Sonia F Vizcarra-Jiménez^{4

5}, Janith P De la Cruz-Galán^{4

6}, Jesús D Gutiérrez-Arratia¹, Blanca Geraldine Quiroga Torres⁴, Alvaro Taype-Rondan^{7

8}

Affiliations

¹ Academic Department, USAMEDIC, Lima, Peru.
² Facultad de Ciencias de la Salud, Carrera de Medicina, Universidad Científica del Sur, Lima, Peru.
³ School of Medicine, Universidad Nacional de Piura, Piura, Peru.
⁴ Comité Permanente Académico, Sociedad Científica Médico Estudiantil Peruana, Lima, Peru.
⁵ Centro de Investigación de Estudiantes de Medicina, Tacna, Peru.
⁶ School of Medicine, Universidad de San Martin de Porres - Filial Norte, Chiclayo, Peru.
⁷ Unidad de Investigación Para la Generación y Síntesis de Evidencias en Salud, Vicerrectorado de Investigación, Universidad San Ignacio de Loyola, Lima, Peru.
⁸ EviSalud - Evidencias en Salud, Lima, Peru.

PMID: 37768724
PMCID: PMC10570896
DOI: 10.2196/48039

Abstract

Background: ChatGPT has shown impressive performance in national medical licensing examinations, such as the United States Medical Licensing Examination (USMLE), even passing it with expert-level performance. However, there is a lack of research on its performance in low-income countries' national licensing medical examinations. In Peru, where almost one out of three examinees fails the national licensing medical examination, ChatGPT has the potential to enhance medical education.

Objective: We aimed to assess the accuracy of ChatGPT using GPT-3.5 and GPT-4 on the Peruvian National Licensing Medical Examination (Examen Nacional de Medicina [ENAM]). Additionally, we sought to identify factors associated with incorrect answers provided by ChatGPT.

Methods: We used the ENAM 2022 data set, which consisted of 180 multiple-choice questions, to evaluate the performance of ChatGPT. Various prompts were used, and accuracy was evaluated. The performance of ChatGPT was compared to that of a sample of 1025 examinees. Factors such as question type, Peruvian-specific knowledge, discrimination, difficulty, quality of questions, and subject were analyzed to determine their influence on incorrect answers. Questions that received incorrect answers underwent a three-step process involving different prompts to explore the potential impact of adding roles and context on ChatGPT's accuracy.

Results: GPT-4 achieved an accuracy of 86% on the ENAM, followed by GPT-3.5 with 77%. The accuracy obtained by the 1025 examinees was 55%. There was a fair agreement (κ=0.38) between GPT-3.5 and GPT-4. Moderate-to-high-difficulty questions were associated with incorrect answers in the crude and adjusted model for GPT-3.5 (odds ratio [OR] 6.6, 95% CI 2.73-15.95) and GPT-4 (OR 33.23, 95% CI 4.3-257.12). After reinputting questions that received incorrect answers, GPT-3.5 went from 41 (100%) to 12 (29%) incorrect answers, and GPT-4 from 25 (100%) to 4 (16%).

Conclusions: Our study found that ChatGPT (GPT-3.5 and GPT-4) can achieve expert-level performance on the ENAM, outperforming most of our examinees. We found fair agreement between both GPT-3.5 and GPT-4. Incorrect answers were associated with the difficulty of questions, which may resemble human performance. Furthermore, by reinputting questions that initially received incorrect answers with different prompts containing additional roles and context, ChatGPT achieved improved accuracy.

Keywords: AI; ChatGPT; ENAM; Examen Nacional de Medicina; Peru; artificial intelligence; assessment; generative pre-trained transformer; learning model; licensing examination; medical education; medical examination.

©Javier A Flores-Cohaila, Abigaíl García-Vicente, Sonia F Vizcarra-Jiménez, Janith P De la Cruz-Galán, Jesús D Gutiérrez-Arratia, Blanca Geraldine Quiroga Torres, Alvaro Taype-Rondan. Originally published in JMIR Medical Education (https://mededu.jmir.org), 28.09.2023.