Generalizability of predictive models for Clostridioides difficile infection, severity and recurrence at an urban safety-net hospital

Abstract

Introduction: Predictive models for Clostridioides difficile infection can identify high-risk patients and aid clinicians in preventing infection. Issues of generalizability regarding current predictive models have been acknowledged but, to the authors' knowledge, have never been quantified.

Methods: C. difficile infection, severity and recurrence predictive models were created using multi-variate logistic regression through case-control sampling from an urban safety-net hospital. Models were validated using five-fold cross-validation, and inverse probability weights (IPW) based on two different catchment area definitions were used to improve external validity. Akaike Information Criterion (AIC), area under the receiver operating characteristic curve (AUROC), and sensitivity and specificity with bootstrapped confidence intervals (CI) were used to assess and compare model fit and performance.

Results: Changes in performance before and after weighting were small across all models, although differences were more apparent after weighting the recurrence model (AUROC values of 0.78, 0.76 and 0.71 for the unweighted and two weighted models, respectively). Overall, the infection model performed the best (AUROC 0.82, 95% CI 0.78-0.85), followed by the recurrence model (AUROC 0.78, 95% CI 0.69-0.86) and then the severity model (AUROC 0.70, 95% CI 0.63-0.78).

Conclusions: The performance of the models after weighting did not change drastically, suggesting that the models predicting C. difficile infection, severity and recurrence may not be impacted by patient selection factors. However, other researchers may wish to consider addressing these catchment forces using IPW.

Keywords: Catchment; Clostridioides difficile; Electronic health record; External validity; Prediction modelling; Urban safety-net hospital.