Development and evaluation of multimodal AI for diagnosis and triage of ophthalmic diseases using ChatGPT and anterior segment images: protocol for a two-stage cross-sectional study

Zhiyu Peng; Ruiqi Ma; Yihan Zhang; Mingxu Yan; Jie Lu; Qian Cheng; Jingjing Liao; Yunqiu Zhang; Jinghan Wang; Yue Zhao; Jiang Zhu; Bing Qin; Qin Jiang; Fei Shi; Jiang Qian; Xinjian Chen; Chen Zhao

doi:10.3389/frai.2023.1323924

Development and evaluation of multimodal AI for diagnosis and triage of ophthalmic diseases using ChatGPT and anterior segment images: protocol for a two-stage cross-sectional study

Front Artif Intell. 2023 Dec 8:6:1323924. doi: 10.3389/frai.2023.1323924. eCollection 2023.

Authors

Zhiyu Peng^#^{1

2

3

4}, Ruiqi Ma^#^{1

3

4}, Yihan Zhang^#^{1

3

4}, Mingxu Yan^#^{1

3

4

5}, Jie Lu^#^{1

3

4

6}, Qian Cheng⁷, Jingjing Liao⁷, Yunqiu Zhang⁶, Jinghan Wang^{1

3

4}, Yue Zhao⁸, Jiang Zhu⁹, Bing Qin⁹, Qin Jiang^{8

10}, Fei Shi⁷, Jiang Qian^{1

3

4}, Xinjian Chen^{7

11}, Chen Zhao^{1

3

4}

Affiliations

¹ Department of Ophthalmology, Fudan Eye & ENT Hospital, Shanghai, China.
² Department of Ophthalmology, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, Zhejiang, China.
³ Laboratory of Myopia, Chinese Academy of Medical Sciences, Shanghai, China.
⁴ NHC Key Laboratory of Myopia, Fudan University, Shanghai, China.
⁵ School of Basic Medical Sciences, Fudan University, Shanghai, China.
⁶ School of Public Health, Fudan University, Shanghai, China.
⁷ Medical Image Processing, Analysis, and Visualization (MIVAP) Lab, School of Electronics and Information Engineering, Soochow University, Suzhou, China.
⁸ The Affiliated Eye Hospital, Nanjing Medical University, Nanjing, China.
⁹ Department of Ophthalmology, Suqian First Hospital, Suqian, China.
¹⁰ The Fourth School of Clinical Medicine, Nanjing Medical University, Nanjing, China.
¹¹ State Key Laboratory of Radiation Medicine and Protection, Soochow University, Suzhou, China.

^# Contributed equally.

Abstract

Introduction: Artificial intelligence (AI) technology has made rapid progress for disease diagnosis and triage. In the field of ophthalmic diseases, image-based diagnosis has achieved high accuracy but still encounters limitations due to the lack of medical history. The emergence of ChatGPT enables human-computer interaction, allowing for the development of a multimodal AI system that integrates interactive text and image information.

Objective: To develop a multimodal AI system using ChatGPT and anterior segment images for diagnosing and triaging ophthalmic diseases. To assess the AI system's performance through a two-stage cross-sectional study, starting with silent evaluation and followed by early clinical evaluation in outpatient clinics.

Methods and analysis: Our study will be conducted across three distinct centers in Shanghai, Nanjing, and Suqian. The development of the smartphone-based multimodal AI system will take place in Shanghai with the goal of achieving ≥90% sensitivity and ≥95% specificity for diagnosing and triaging ophthalmic diseases. The first stage of the cross-sectional study will explore the system's performance in Shanghai's outpatient clinics. Medical histories will be collected without patient interaction, and anterior segment images will be captured using slit lamp equipment. This stage aims for ≥85% sensitivity and ≥95% specificity with a sample size of 100 patients. The second stage will take place at three locations, with Shanghai serving as the internal validation dataset, and Nanjing and Suqian as the external validation dataset. Medical history will be collected through patient interviews, and anterior segment images will be captured via smartphone devices. An expert panel will establish reference standards and assess AI accuracy for diagnosis and triage throughout all stages. A one-vs.-rest strategy will be used for data analysis, and a post-hoc power calculation will be performed to evaluate the impact of disease types on AI performance.

Discussion: Our study may provide a user-friendly smartphone-based multimodal AI system for diagnosis and triage of ophthalmic diseases. This innovative system may support early detection of ocular abnormalities, facilitate establishment of a tiered healthcare system, and reduce the burdens on tertiary facilities.

Trial registration: The study was registered in ClinicalTrials.gov on June 25th, 2023 (NCT05930444).

Keywords: ChatGPT; anterior segment images; cross-sectional study; multimodal AI; ophthalmic diseases; prompt engineering; protocol.

Associated data

ClinicalTrials.gov/NCT05930444

Grants and funding

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This work was supported by the National Natural Science Foundation of China (Grant Numbers 82371101, 82000940, 81970835, 81800867, 82020108006, and 81730025), the Natural Science Foundation of Shanghai (Grant Number 20ZR1409500), Excellent Academic Leaders of Shanghai (Grant Number 18XD1401000); The Open Research Funds of the State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University (Grant Number 303060202400375).