Performance of AI chatbots on controversial topics in oral medicine, pathology, and radiology

Hossein Mohammad-Rahimi; Zaid H Khoury; Mina Iranparvar Alamdari; Rata Rokhshad; Parisa Motie; Azin Parsa; Tiffany Tavares; James J Sciubba; Jeffery B Price; Ahmed S Sultan

doi:10.1016/j.oooo.2024.01.015

Performance of AI chatbots on controversial topics in oral medicine, pathology, and radiology

Oral Surg Oral Med Oral Pathol Oral Radiol. 2024 May;137(5):508-514. doi: 10.1016/j.oooo.2024.01.015. Epub 2024 Feb 6.

Authors

Affiliations

¹ Division of Artificial Intelligence Research, University of Maryland School of Dentistry, Baltimore, MD, USA; Topic Group Dental Diagnostics and Digital Dentistry, ITU/WHO Focus Group AI on Health, Berlin, Germany.
² Department of Oral Diagnostic Sciences and Research, Meharry Medical College School of Dentistry, Nashville, TN, USA.
³ Department of Oral and Maxillofacial Radiology, School of Dentistry, Shahid Beheshti University of Medical Sciences, Tehran, Iran.
⁴ Topic Group Dental Diagnostics and Digital Dentistry, ITU/WHO Focus Group AI on Health, Berlin, Germany.
⁵ Medical Image and Signal Processing Research Center, Medical University of Isfahan, Isfahan, Iran.
⁶ Department of Oncology and Diagnostic Sciences, University of Maryland School of Dentistry, Baltimore, MD, USA.
⁷ Department of Comprehensive Dentistry, UT Health San Antonio School of Dentistry, San Antonio, TX, USA.
⁸ Department of Otolaryngology, Head & Neck Surgery, The Johns Hopkins University, Baltimore, MD, USA.
⁹ Division of Artificial Intelligence Research, University of Maryland School of Dentistry, Baltimore, MD, USA; Department of Oncology and Diagnostic Sciences, University of Maryland School of Dentistry, Baltimore, MD, USA.
¹⁰ Division of Artificial Intelligence Research, University of Maryland School of Dentistry, Baltimore, MD, USA; Department of Oncology and Diagnostic Sciences, University of Maryland School of Dentistry, Baltimore, MD, USA; University of Maryland Marlene and Stewart Greenebaum Comprehensive Cancer Center, Baltimore, MD, USA. Electronic address: asultan1@umaryland.edu.

PMID: 38553304
DOI: 10.1016/j.oooo.2024.01.015

Abstract

Objectives: In this study, we assessed 6 different artificial intelligence (AI) chatbots (Bing, GPT-3.5, GPT-4, Google Bard, Claude, Sage) responses to controversial and difficult questions in oral pathology, oral medicine, and oral radiology.

Study design: The chatbots' answers were evaluated by board-certified specialists using a modified version of the global quality score on a 5-point Likert scale. The quality and validity of chatbot citations were evaluated.

Results: Claude had the highest mean score of 4.341 ± 0.582 for oral pathology and medicine. Bing had the lowest scores of 3.447 ± 0.566. In oral radiology, GPT-4 had the highest mean score of 3.621 ± 1.009 and Bing the lowest score of 2.379 ± 0.978. GPT-4 achieved the highest mean score of 4.066 ± 0.825 for performance across all disciplines. 82 out of 349 (23.50%) of generated citations from chatbots were fake.

Conclusions: The most superior chatbot in providing high-quality information for controversial topics in various dental disciplines was GPT-4. Although the majority of chatbots performed well, it is suggested that developers of AI medical chatbots incorporate scientific citation authenticators to validate the outputted citations given the relatively high number of fabricated citations.

MeSH terms

Artificial Intelligence*
Humans
Oral Medicine*
Pathology, Oral
Radiology