Improving ICU Risk Predictive Models Through Automation Designed for Resiliency Against Documentation Bias

Abstract

Objectives: Electronic health records enable automated data capture for risk models but may introduce bias. We present the Philips Critical Care Outcome Prediction Model (CCOPM) focused on addressing model features sensitive to data drift to improve benchmarking ICUs on mortality performance.

Design: Retrospective, multicenter study of ICU patients randomized in 3:2 fashion into development and validation cohorts. Generalized additive models (GAM) with features designed to mitigate biases introduced from documentation of admission diagnosis, Glasgow Coma Scale (GCS), and extreme vital signs were developed using clinical features representing the first 24 hours of ICU admission.

Setting: eICU Research Institute database derived from ICUs participating in the Philips eICU telecritical care program.

Patients: A total of 572,985 adult ICU stays discharged from the hospital between January 1, 2017, and December 31, 2018, were included, yielding 509,586 stays in the final cohort; 305,590 and 203,996 in development and validation cohorts, respectively.

Interventions: None.

Measurements and main results: Model discrimination was compared against Acute Physiology and Chronic Health Evaluation (APACHE) IVa/IVb models on the validation cohort using the area under the receiver operating characteristic (AUROC) curve. Calibration assessed by actual/predicted ratios, calibration-in-the-large statistics, and visual analysis. Performance metrics were further stratified by subgroups of admission diagnosis and ICU characteristics. Historic data from two health systems with abrupt changes in Glasgow Coma Scale (GCS) documentation were assessed in the year prior to and after data shift. CCOPM outperformed APACHE IVa/IVb for ICU mortality (AUROC, 0.925 vs 0.88) and hospital mortality (AUROC, 0.90 vs 0.86). Better calibration performance was also attained among subgroups of different admission diagnoses, ICU types, and over unique ICU-years. The CCOPM provided more stable predictions compared with APACHE IVa within an external cohort of greater than 120,000 patients from two health systems with known changes in GCS documentation.

Conclusions: These mortality risk models demonstrated excellent performance compared with APACHE while appearing to mitigate bias introduced through major shifts in GCS documentation at two large health systems. This provides evidence to support using automated capture rather than trained personnel for capture of GCS data used in benchmarking ICUs on mortality performance.