Purpose: ChatGPT (Chat-Generative Pre-trained Transformer) has proven to be a powerful information tool on various topics, including healthcare. This system is based on information obtained on the Internet, but this information is not always reliable. Currently, few studies analyze the validity of these responses in rhinology. Our work aims to assess the quality and reliability of the information provided by AI regarding the main rhinological pathologies.
Methods: We asked to the default ChatGPT version (GPT-3.5) 65 questions about the most prevalent pathologies in rhinology. The focus was learning about the causes, risk factors, treatments, prognosis, and outcomes. We use the Discern questionnaire and a hexagonal radar schema to evaluate the quality of the information. We use Fleiss's kappa statistical analysis to determine the consistency of agreement between different observers.
Results: The overall evaluation of the Discern questionnaire resulted in a score of 4.05 (± 0.6). The results in the Reliability section are worse, with an average score of 3.18. (± 1.77). This score is affected by the responses to questions about the source of the information provided. The average score for the Quality section was 3.59 (± 1.18). Fleiss's Kappa shows substantial agreement, with a K of 0.69 (p < 0.001).
Conclusion: The ChatGPT answers are accurate and reliable. It generates a simple and understandable description of the pathology for the patient's benefit. Our team considers that ChatGPT could be a useful tool to provide information under prior supervision by a health professional.
Keywords: Artificial intelligence; ChatGPT; Chatbot; Healthcare; Natural language processing; Rhinology.
© 2024. The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature.