Assessing the accuracy and consistency of ChatGPT in clinical pharmacy management: A preliminary analysis with clinical pharmacy experts worldwide

Zahraa Al-Dujaili; Sarah Omari; Jey Pillai; Achraf Al Faraj

doi:10.1016/j.sapharm.2023.08.012

Assessing the accuracy and consistency of ChatGPT in clinical pharmacy management: A preliminary analysis with clinical pharmacy experts worldwide

Res Social Adm Pharm. 2023 Dec;19(12):1590-1594. doi: 10.1016/j.sapharm.2023.08.012. Epub 2023 Sep 7.

Authors

Zahraa Al-Dujaili¹, Sarah Omari², Jey Pillai¹, Achraf Al Faraj³

Affiliations

¹ College of Pharmacy, American University of Iraq - Baghdad (AUIB), Baghdad, Iraq.
² Department of Epidemiology and Population Health, American University of Beirut (AUB), Beirut, Lebanon.
³ College of Pharmacy, American University of Iraq - Baghdad (AUIB), Baghdad, Iraq. Electronic address: achraf.alfaraj@gmail.com.

PMID: 37696742
DOI: 10.1016/j.sapharm.2023.08.012

Abstract

Background: ChatGPT conversation system has ushered in a revolutionary new era of information retrieval and stands as one of the fastest-growing platforms. Clinical pharmacy, as a dynamic discipline, necessitates an advanced comprehension of drugs and diseases. The process of decision-making in clinical pharmacy demands accuracy and consistency in medical information, as it directly affects patient safety.

Objective: The objective was to evaluate ChatGPT's accuracy and consistency in managing pharmacotherapy cases across multiple time points. Additionally, input was gathered from global clinical pharmacy experts, and the agreement between ChatGPT's responses and those of clinical pharmacy experts worldwide was assessed.

Methods: A set of 20 cases of pharmacotherapy was entered into ChatGPT at three different time points. Reliability analysis was performed using inter-rater reliability to measure the accuracy of the output generated by ChatGPT at each time point. Test-retest reliability was performed to measure the consistency of the output generated by ChatGPT across the three time points. Pharmacy expert performance was evaluated, and the overall results were compared.

Results: ChatGPT achieved a hit rate of 70.83% at week 1, 79.2% at week 3, and 75% at week 5. The percent agreement between weeks 1 and 3 was 79.2%, whereas it was 87.5% between weeks 3 and 5, and 83.3% between weeks 1 and 5. In contrast, accuracy rates among clinical pharmacy experts showed considerable variation according to their geographic location. The highest agreement between clinical pharmacist responses and ChatGPT responses was observed at the last time point examined.

Conclusions: Overall, the analysis suggested that ChatGPT is capable of generating clinically relevant pharmaceutical information, albeit with some variation in accuracy and consistency. It should be noted that clinical pharmacy experts worldwide may provide varying degrees of accuracy depending on their expertise. This study highlights the potential of AI chatbots in clinical pharmacy.

Keywords: AI chatbot; ChatGPT; Clinical pharmacy; Decision making; Pharmacotherapy.