Predicting COVID-19 severity: Challenges in reproducibility and deployment of machine learning methods

Luwei Liu; Wenyu Song; Namrata Patil; Michael Sainlaire; Ravi Jasuja; Patricia C Dykes

doi:10.1016/j.ijmedinf.2023.105210

Predicting COVID-19 severity: Challenges in reproducibility and deployment of machine learning methods

Int J Med Inform. 2023 Nov:179:105210. doi: 10.1016/j.ijmedinf.2023.105210. Epub 2023 Sep 7.

Authors

Luwei Liu¹, Wenyu Song², Namrata Patil³, Michael Sainlaire¹, Ravi Jasuja⁴, Patricia C Dykes²

Affiliations

¹ Department of Medicine, Brigham & Women's Hospital, Boston, MA, USA.
² Department of Medicine, Brigham & Women's Hospital, Boston, MA, USA; Harvard Medical School, Boston, MA, USA.
³ Department of Surgery, Brigham & Women's Hospital, Boston, MA, USA; Harvard Medical School, Boston, MA, USA.
⁴ Department of Medicine, Brigham & Women's Hospital, Boston, MA, USA; Harvard Medical School, Boston, MA, USA. Electronic address: rjasuja@bwh.harvard.edu.

PMID: 37769368
DOI: 10.1016/j.ijmedinf.2023.105210

Abstract

The increasing use of electronic health records (EHR) based computable phenotypes in clinical research is providing new opportunities for development of data-driven medical applications. Adopted widely in the United States and globally, EHRs facilitate systematic collection of patients' longitudinal information, which serves as one of the important foundations for artificial intelligence applications in medicine. Harmonization of input variables and outcome definitions is critically important for wider clinical applicability of artificial intelligence (AI) methodologies. In this review, we focused on Coronavirus Disease 2019 (COVID-19) severity machine learning prediction models and explored the pipeline for standardizing future disease severity model development using EHR information. We identified 2,967 studies published between 01/01/2020 and 02/15/2022 and selected 135 independent studies that had built machine learning prediction models to predict severity related outcomes of COVID-19 patients based on EHR data for the final review. These 135 studies spanning across 27 counties covered a broad range of severity related prediction outcomes. We observed substantial inconsistency in COVID-19 severity phenotype definitions among models in these studies. Moreover, there was a gap between the outcome of these models and clinician-recognized clinical concepts. Accordingly, we recommend that robust clinical input metrics, with outcome definitions which eliminate ambiguity in interpretation, to reduce algorithmic bias, mitigate model brittleness and improve generalizability of a universal model for COVID-19 severity. This framework can potentially be extended to broader clinical application.

Keywords: Artificial Intelligence; COVID-19 Severity; Electronic Health Record; Machine Learning; Phenotype.

Publication types

Review