Diagnostic Accuracy and Failure Mode Analysis of a Deep Learning Algorithm for the Detection of Intracranial Hemorrhage

Andrew F Voter; Ece Meram; John W Garrett; John-Paul J Yu

doi:10.1016/j.jacr.2021.03.005

Diagnostic Accuracy and Failure Mode Analysis of a Deep Learning Algorithm for the Detection of Intracranial Hemorrhage

J Am Coll Radiol. 2021 Aug;18(8):1143-1152. doi: 10.1016/j.jacr.2021.03.005. Epub 2021 Apr 3.

Authors

Andrew F Voter¹, Ece Meram², John W Garrett², John-Paul J Yu³

Affiliations

¹ School of Medicine and Public Health, University of Wisconsin-Madison, Madison, Wisconsin.
² Department of Radiology, University of Wisconsin-Madison, Madison, Wisconsin.
³ Department of Radiology, University of Wisconsin-Madison, Madison, Wisconsin; Department of Biomedical Engineering, College of Engineering, University of Wisconsin-Madison, Madison, Wisconsin; Department of Psychiatry, University of Wisconsin School of Medicine and Public Health, Madison, Wisconsin. Electronic address: jpyu@uwhealth.org.

Abstract

Objective: To determine the institutional diagnostic accuracy of an artificial intelligence (AI) decision support systems (DSS), Aidoc, in diagnosing intracranial hemorrhage (ICH) on noncontrast head CTs and to assess the potential generalizability of an AI DSS.

Methods: This retrospective study included 3,605 consecutive, emergent, adult noncontrast head CT scans performed between July 1, 2019, and December 30, 2019, at our institution (51% female subjects, mean age of 61 ± 21 years). Each scan was evaluated for ICH by both a certificate of added qualification certified neuroradiologist and Aidoc. We determined the diagnostic accuracy of the AI model and performed a failure mode analysis with quantitative CT radiomic image characterization.

Results: Of the 3,605 scans, 349 cases of ICH (9.7% of studies) were identified. The neuroradiologist and Aidoc interpretations were concordant in 96.9% of cases and the overall sensitivity, specificity, positive predictive value, and negative predictive value were 92.3%, 97.7%, 81.3%, and 99.2%, respectively, with positive predictive values unexpectedly lower than in previously reported studies. Prior neurosurgery, type of ICH, and number of ICHs were significantly associated with decreased model performance. Quantitative image characterization with CT radiomics failed to reveal significant differences between concordant and discordant studies.

Discussion: This study revealed decreased diagnostic accuracy of an AI DSS at our institution. Despite extensive evaluation, we were unable to identify the source of this discrepancy, raising concerns about the generalizability of these tools with indeterminate failure modes. These results further highlight the need for standardized study design to allow for rigorous and reproducible site-to-site comparison of emerging deep learning technologies.

Keywords: Artificial intelligence; decision support systems; generalizability; intracranial hemorrhage; noncontrast head CT.

MeSH terms

Adult
Aged
Aged, 80 and over
Algorithms
Artificial Intelligence*
Deep Learning*
Female
Humans
Intracranial Hemorrhages / diagnostic imaging
Male
Middle Aged
Retrospective Studies

Abstract

MeSH terms

Grants and funding