Examining Real-World Medication Consultations and Drug-Herb Interactions: ChatGPT Performance Evaluation

Hsing-Yu Hsu; Kai-Cheng Hsu; Shih-Yen Hou; Ching-Lung Wu; Yow-Wen Hsieh; Yih-Dih Cheng

doi:10.2196/48433

Examining Real-World Medication Consultations and Drug-Herb Interactions: ChatGPT Performance Evaluation

JMIR Med Educ. 2023 Aug 21:9:e48433. doi: 10.2196/48433.

Authors

Hsing-Yu Hsu^#^{1

2}, Kai-Cheng Hsu^#^{3

4}, Shih-Yen Hou³, Ching-Lung Wu⁵, Yow-Wen Hsieh^#^{1

5}, Yih-Dih Cheng^#^{1

5}

Affiliations

¹ Department of Pharmacy, China Medical University Hospital, Taichung, Taiwan.
² Graduate Institute of Clinical Pharmacy, College of Medicine, National Taiwan University, Taipei, Taiwan.
³ Artificial Intelligence Center, China Medical University Hospital, Taichung, Taiwan.
⁴ Department of Medicine, China Medical University, Taichung, Taiwan.
⁵ School of Pharmacy, College of Pharmacy, China Medical University, Taichung, Taiwan.

^# Contributed equally.

PMID: 37561097
PMCID: PMC10477918
DOI: 10.2196/48433

Abstract

Background: Since OpenAI released ChatGPT, with its strong capability in handling natural tasks and its user-friendly interface, it has garnered significant attention.

Objective: A prospective analysis is required to evaluate the accuracy and appropriateness of medication consultation responses generated by ChatGPT.

Methods: A prospective cross-sectional study was conducted by the pharmacy department of a medical center in Taiwan. The test data set comprised retrospective medication consultation questions collected from February 1, 2023, to February 28, 2023, along with common questions about drug-herb interactions. Two distinct sets of questions were tested: real-world medication consultation questions and common questions about interactions between traditional Chinese and Western medicines. We used the conventional double-review mechanism. The appropriateness of each response from ChatGPT was assessed by 2 experienced pharmacists. In the event of a discrepancy between the assessments, a third pharmacist stepped in to make the final decision.

Results: Of 293 real-world medication consultation questions, a random selection of 80 was used to evaluate ChatGPT's performance. ChatGPT exhibited a higher appropriateness rate in responding to public medication consultation questions compared to those asked by health care providers in a hospital setting (31/51, 61% vs 20/51, 39%; P=.01).

Conclusions: The findings from this study suggest that ChatGPT could potentially be used for answering basic medication consultation questions. Our analysis of the erroneous information allowed us to identify potential medical risks associated with certain questions; this problem deserves our close attention.

Keywords: ChatGPT; LLM; NLP; chat generative pre-trained transformer; drug-herb interactions; language models; large language model; natural language processing; pharmacist; real-world medication consultation questions.

©Hsing-Yu Hsu, Kai-Cheng Hsu, Shih-Yen Hou, Ching-Lung Wu, Yow-Wen Hsieh, Yih-Dih Cheng. Originally published in JMIR Medical Education (https://mededu.jmir.org), 21.08.2023.