Generalizability of Risk Stratification Algorithms for Exacerbations in COPD

Abstract

Background: Contemporary management of COPD relies on exacerbation history to risk-stratify patients for future exacerbations. Multivariable prediction models can improve the performance of risk stratification. However, the clinical utility of risk stratification can vary from one population to another.

Research question: How do two validated exacerbation risk prediction models (Acute COPD Exacerbation Prediction Tool [ACCEPT] and the Bertens model) compared with exacerbation history alone perform in different patient populations?

Study design and methods: We used data from three clinical studies representing populations at different levels of moderate to severe exacerbation risk: the Study to Understand Mortality and Morbidity in COPD (SUMMIT; N = 2,421; annual risk, 0.22), the Long-term Oxygen Treatment Trial (LOTT; N = 595; annual risk, 0.38), and Towards a Revolution in COPD Health (TORCH; N = 1,091; annual risk, 0.52). We compared the area under the receiver operating characteristic curve (AUC) and net benefit (measure of clinical utility) among three risk stratification algorithms for predicting exacerbations in the next 12 months. We also evaluated the effect of model recalibration on clinical utility.

Results: Compared with exacerbation history, ACCEPT showed better performance in all three samples (change in AUC, 0.08, 0.07, and 0.10, in SUMMIT, LOTT, and TORCH, respectively; P ≤ .001 for all). The Bertens model showed better performance compared with exacerbation history in SUMMIT and TORCH (change in AUC, 0.10 and 0.05, respectively; P < .001 for both), but not in LOTT. No algorithm was superior in clinical utility across all samples. Before recalibration, the Bertens model generally outperformed the other algorithms in low-risk settings, whereas ACCEPT outperformed others in high-risk settings. All three algorithms showed the risk of harm (providing lower net benefit than not using any risk stratification). After recalibration, risk of harm was mitigated substantially for both prediction models.

Interpretation: Exacerbation history alone is unlikely to provide clinical utility for predicting COPD exacerbations in all settings and could be associated with a risk of harm. Prediction models have superior predictive performance, but require setting-specific recalibration to confer higher clinical utility.

Keywords: COPD; clinical prediction modeling; clinical utility; precision medicine; risk stratification.