Medical knowledge of ChatGPT in public health, infectious diseases, COVID-19 pandemic, and vaccines: multiple choice questions examination based performance

Sultan Ayoub Meo; Metib Alotaibi; Muhammad Zain Sultan Meo; Muhammad Omair Sultan Meo; Mashhood Hamid

doi:10.3389/fpubh.2024.1360597

Medical knowledge of ChatGPT in public health, infectious diseases, COVID-19 pandemic, and vaccines: multiple choice questions examination based performance

Front Public Health. 2024 Apr 17:12:1360597. doi: 10.3389/fpubh.2024.1360597. eCollection 2024.

Authors

Sultan Ayoub Meo¹, Metib Alotaibi², Muhammad Zain Sultan Meo³, Muhammad Omair Sultan Meo³, Mashhood Hamid⁴

Affiliations

¹ Department of Physiology, College of Medicine, King Saud University, Riyadh, Saudi Arabia.
² Department of Medicine, College of Medicine, King Saud University, Riyadh, Saudi Arabia.
³ College of Medicine, Alfaisal University, Riyadh, Saudi Arabia.
⁴ Department of Family and Community Medicine, College of Medicine, King Saud University, Riyadh, Saudi Arabia.

Abstract

Background: At the beginning of the year 2023, the Chatbot Generative Pre-Trained Transformer (ChatGPT) gained remarkable attention from the public. There is a great discussion about ChatGPT and its knowledge in medical sciences, however, literature is lacking to evaluate the ChatGPT knowledge level in public health. Therefore, this study investigates the knowledge of ChatGPT in public health, infectious diseases, the COVID-19 pandemic, and its vaccines.

Methods: Multiple Choice Questions (MCQs) bank was established. The question's contents were reviewed and confirmed that the questions were appropriate to the contents. The MCQs were based on the case scenario, with four sub-stems, with a single correct answer. From the MCQs bank, 60 MCQs we selected, 30 MCQs were from public health, and infectious diseases topics, 17 MCQs were from the COVID-19 pandemic, and 13 MCQs were on COVID-19 vaccines. Each MCQ was manually entered, and tasks were given to determine the knowledge level of ChatGPT on MCQs.

Results: Out of a total of 60 MCQs in public health, infectious diseases, the COVID-19 pandemic, and vaccines, ChatGPT attempted all the MCQs and obtained 17/30 (56.66%) marks in public health, infectious diseases, 15/17 (88.23%) in COVID-19, and 12/13 (92.30%) marks in COVID-19 vaccines MCQs, with an overall score of 44/60 (73.33%). The observed results of the correct answers in each section were significantly higher (p = 0.001). The ChatGPT obtained satisfactory grades in all three domains of public health, infectious diseases, and COVID-19 pandemic-allied examination.

Conclusion: ChatGPT has satisfactory knowledge of public health, infectious diseases, the COVID-19 pandemic, and its vaccines. In future, ChatGPT may assist medical educators, academicians, and healthcare professionals in providing a better understanding of public health, infectious diseases, the COVID-19 pandemic, and vaccines.

Keywords: ChatGPT; artificial intelligence; infectious disease; knowledge; medical education; public health.

Publication types

Research Support, Non-U.S. Gov't

MeSH terms

COVID-19 Vaccines*
COVID-19* / epidemiology
COVID-19* / prevention & control
Communicable Diseases / epidemiology
Educational Measurement
Health Knowledge, Attitudes, Practice
Humans
Pandemics / prevention & control
Public Health*
SARS-CoV-2
Surveys and Questionnaires

Substances

COVID-19 Vaccines

Grants and funding

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This study was supported by Deputyship for Research and Innovation, Ministry of Education in Saudi Arabia (IFKSUOR3-4-6).