ChatGPT accurately performs genetic counseling for gynecologic cancers

Jharna M Patel; Catherine E Hermann; Whitfield B Growdon; Emeline Aviki; Marina Stasenko

doi:10.1016/j.ygyno.2024.04.006

ChatGPT accurately performs genetic counseling for gynecologic cancers

Gynecol Oncol. 2024 Apr 26:183:115-119. doi: 10.1016/j.ygyno.2024.04.006. Online ahead of print.

Authors

Jharna M Patel¹, Catherine E Hermann², Whitfield B Growdon², Emeline Aviki³, Marina Stasenko²

Affiliations

¹ New York University Langone Health, Department of Obstetrics and Gynecology, Division of Gynecologic Oncology, New York, NY, United States of America. Electronic address: Jharna.Patel@nyulangone.org.
² New York University Langone Health, Department of Obstetrics and Gynecology, Division of Gynecologic Oncology, New York, NY, United States of America.
³ New York University Langone Health, Long Island, Department of Obstetrics and Gynecology, Division of Gynecologic Oncology, Mineola, NY, United States of America.

PMID: 38676973
DOI: 10.1016/j.ygyno.2024.04.006

Abstract

Objective: Artificial Intelligence (AI) systems such as ChatGPT can take medical examinations and counsel patients regarding medical diagnosis. We aim to quantify the accuracy of the ChatGPT V3.4 in answering commonly asked questions pertaining to genetic testing and counseling for gynecologic cancers.

Methods: Forty questions were formulated in conjunction with gynecologic oncologists and adapted from professional society guidelines and ChatGPT version 3.5 was queried, the version that is readily available to the public. The two categories of questions were genetic counseling guidelines and questions pertaining to specific genetic disorders. The answers were scored by two attending Gynecologic Oncologists according to the following scale: 1) correct and comprehensive, 2) correct but not comprehensive, 3) some correct, some incorrect, and 4) completely incorrect. Scoring discrepancies were resolved by additional third reviewer. The proportion of responses earning each score were calculated overall and within each question category.

Results: ChatGPT provided correct and comprehensive answers to 33/40 (82.5%) questions, correct but not comprehensive answers to 6/40 (15%) questions, partially incorrect answers to 1/40 (2.5%) questions, and completely incorrect answers to 0/40 (0%) questions. The genetic counseling category of questions had the highest proportion of answers that were both correct and comprehensive with ChatGPT answering all 20/20 questions with 100% accuracy and were comprehensive in responses. ChatGPT performed equally in the specific genetic disorders category, with 88.2% (15/17) and 66.6% (2/3) correct and comprehensive answers to questions pertaining to hereditary breast and ovarian cancer and Lynch syndrome questions respectively.

Conclusion: ChatGPT accurately answers questions about genetic syndromes, genetic testing, and counseling in majority of the studied questions. These data suggest this powerful tool can be utilized as a patient resource for genetic counseling questions, though more data input from gynecologic oncologists would be needed to educate patients on genetic syndromes.

Keywords: ChatGPT; Genetic counseling; Genetic testing.