ChatGPT-3.5 System Usability Scale early assessment among Healthcare Workers: Horizons of adoption in medical practice

Fadi Aljamaan; Khalid H Malki; Khalid Alhasan; Amr Jamal; Ibraheem Altamimi; Afnan Khayat; Ali Alhaboob; Naif Abdulmajeed; Fatimah S Alshahrani; Khaled Saad; Ayman Al-Eyadhy; Jaffar A Al-Tawfiq; Mohamad-Hani Temsah

doi:10.1016/j.heliyon.2024.e28962

ChatGPT-3.5 System Usability Scale early assessment among Healthcare Workers: Horizons of adoption in medical practice

Heliyon. 2024 Apr 7;10(7):e28962. doi: 10.1016/j.heliyon.2024.e28962. eCollection 2024 Apr 15.

Authors

Fadi Aljamaan^{1

2}, Khalid H Malki³, Khalid Alhasan^{1

4

5}, Amr Jamal^{1

6

7}, Ibraheem Altamimi¹, Afnan Khayat⁸, Ali Alhaboob^{1

4}, Naif Abdulmajeed^{4

9}, Fatimah S Alshahrani^{1

10}, Khaled Saad¹¹, Ayman Al-Eyadhy^{1

4}, Jaffar A Al-Tawfiq^{12

13

14}, Mohamad-Hani Temsah^{1

4

7}

Affiliations

¹ College of Medicine, King Saud University, Riyadh 11362, Saudi Arabia.
² Critical Care Department, College of Medicine, King Saud University, Riyadh 11362, Saudi Arabia.
³ Research Chair of Voice, Swallowing, and Communication Disorders, Department of Otolaryngology, College of Medicine, King Saud University, Riyadh 11362, Saudi Arabia.
⁴ Pediatric Department, College of Medicine, King Saud University Medical City, Riyadh 11362, Saudi Arabia.
⁵ Department of Kidney and Pancreas Transplant, Organ Transplant Center of Excellence, King Faisal Specialist Hospital and Research Center, Riyadh 11211, Saudi Arabia.
⁶ Department of Family and Community Medicine, King Saud University Medical City, Riyadh 11362, Saudi Arabia.
⁷ Evidence-Based Health Care & Knowledge Translation Research Chair, Family & Community Medicine Department, College of Medicine, King Saud University, Riyadh 11362, Saudi Arabia.
⁸ Health Information Management Department, Prince Sultan Military College of Health Sciences, Al Dhahran 34313, Saudi Arabia.
⁹ Pediatric Nephrology Department, Prince Sultan Military Medical City, Riyadh 11159, Saudi Arabia.
¹⁰ Infectious Disease Division, Department of Medicine, King Saud University Medical City, Riyadh 11362, Saudi Arabia.
¹¹ Pediatric Department, Faculty of Medicine, Assiut University, Assiut 71516, Egypt.
¹² Specialty Internal Medicine and Quality Department, Johns Hopkins Aramco Healthcare, Dhahran 34465, Saudi Arabia.
¹³ Infectious Disease Division, Department of Medicine, Indiana University School of Medicine, Indianapolis, IN46202, USA.
¹⁴ Infectious Disease Division, Department of Medicine, Johns Hopkins University School of Medicine, Baltimore, MD21218, USA.

Abstract

Artificial intelligence (AI) chatbots, such as ChatGPT, have widely invaded all domains of human life. They have the potential to transform healthcare future. However, their effective implementation hinges on healthcare workers' (HCWs) adoption and perceptions. This study aimed to evaluate HCWs usability of ChatGPT three months post-launch in Saudi Arabia using the System Usability Scale (SUS). A total of 194 HCWs participated in the survey. Forty-seven percent were satisfied with their usage, 57 % expressed moderate to high trust in its ability to generate medical decisions. 58 % expected ChatGPT would improve patients' outcomes, even though 84 % were optimistic of its potential to improve the future of healthcare practice. They expressed possible concerns like recommending harmful medical decisions and medicolegal implications. The overall mean SUS score was 64.52, equivalent to 50 % percentile rank, indicating high marginal acceptability of the system. The strongest positive predictors of high SUS scores were participants' belief in AI chatbot's benefits in medical research, self-rated familiarity with ChatGPT and self-rated computer skills proficiency. Participants' learnability and ease of use score correlated positively but weakly. On the other hand, medical students and interns had significantly high learnability scores compared to others, while ease of use scores correlated very strongly with participants' perception of positive impact of ChatGPT on the future of healthcare practice. Our findings highlight the HCWs' perceived marginal acceptance of ChatGPT at the current stage and their optimism of its potential in supporting them in future practice, especially in the research domain, in addition to humble ambition of its potential to improve patients' outcomes particularly in regard of medical decisions. On the other end, it underscores the need for ongoing efforts to build trust and address ethical and legal concerns of AI implications in healthcare. The study contributes to the growing body of literature on AI chatbots in healthcare, especially addressing its future improvement strategies and provides insights for policymakers and healthcare providers about the potential benefits and challenges of implementing them in their practice.

Keywords: AI in healthcare; Artificial Intelligence; ChatGPT; Chatbots in medical practice; Digital health; Healthcare technology; Healthcare workers usability; Saudi Arabia; System Usability Scale.