Characteristics of a Large, Labeled Data Set for the Training of Artificial Intelligence for Glaucoma Screening with Fundus Photographs

Hans G Lemij; Coen de Vente; Clara I Sánchez; Koen A Vermeer

doi:10.1016/j.xops.2023.100300

Characteristics of a Large, Labeled Data Set for the Training of Artificial Intelligence for Glaucoma Screening with Fundus Photographs

Ophthalmol Sci. 2023 Mar 17;3(3):100300. doi: 10.1016/j.xops.2023.100300. eCollection 2023 Sep.

Authors

Hans G Lemij¹, Coen de Vente^{2

3}, Clara I Sánchez^{2

3}, Koen A Vermeer⁴

Affiliations

¹ The Rotterdam Eye Hospital, Rotterdam, the Netherlands.
² Quantitative Healthcare Analysis (QurAI) Group, Informatics Institute, University of Amsterdam, Amsterdam, the Netherlands.
³ Department of Biomedical Engineering and Physics, Amsterdam UMC, Amsterdam, the Netherlands.
⁴ The Rotterdam Ophthalmic Institute, Rotterdam Eye Hospital, Rotterdam, the Netherlands.

Abstract

Purpose: Significant visual impairment due to glaucoma is largely caused by the disease being detected too late.

Objective: To build a labeled data set for training artificial intelligence (AI) algorithms for glaucoma screening by fundus photography, to assess the accuracy of the graders, and to characterize the features of all eyes with referable glaucoma (RG).

Design: Cross-sectional study.

Subjects: Color fundus photographs (CFPs) of 113 893 eyes of 60 357 individuals were obtained from EyePACS, California, United States, from a population screening program for diabetic retinopathy.

Methods: Carefully selected graders (ophthalmologists and optometrists) graded the images. To qualify, they had to pass the European Optic Disc Assessment Trial optic disc assessment with ≥ 85% accuracy and 92% specificity. Of 90 candidates, 30 passed. Each image of the EyePACS set was then scored by varying random pairs of graders as "RG," "no referable glaucoma (NRG)," or "ungradable (UG)." In case of disagreement, a glaucoma specialist made the final grading. Referable glaucoma was scored if visual field damage was expected. In case of RG, graders were instructed to mark up to 10 relevant glaucomatous features.

Main outcome measures: Qualitative features in eyes with RG.

Results: The performance of each grader was monitored; if the sensitivity and specificity dropped below 80% and 95%, respectively (the final grade served as reference), they exited the study and their gradings were redone by other graders. In all, 20 graders qualified; their mean sensitivity and specificity (standard deviation [SD]) were 85.6% (5.7) and 96.1% (2.8), respectively. The 2 graders agreed in 92.45% of the images (Gwet's AC2, expressing the inter-rater reliability, was 0.917). Of all gradings, the sensitivity and specificity (95% confidence interval) were 86.0 (85.2-86.7)% and 96.4 (96.3-96.5)%, respectively. Of all gradable eyes (n = 111 183; 97.62%) the prevalence of RG was 4.38%. The most common features of RG were the appearance of the neuroretinal rim (NRR) inferiorly and superiorly.

Conclusions: A large data set of CFPs was put together of sufficient quality to develop AI screening solutions for glaucoma. The most common features of RG were the appearance of the NRR inferiorly and superiorly. Disc hemorrhages were a rare feature of RG.

Financial disclosures: Proprietary or commercial disclosure may be found after the references.

Keywords: Artificial intelligence; Clinical features; color fundus photographs; glaucoma screening; labeled data set.