Background: The clinical value of a prognostic score depends on its out-of-sample validity because inaccurate outcome prediction can be not only useless but potentially fatal. We aimed to evaluate the out-of-sample validity of a recently developed and highly accurate Korean prognostic score for predicting neurologic outcome after cardiac arrest in an independent, plausibly related sample of European cardiac arrest survivors.
Methods: Analysis of data from a European cardiac arrest center, certified in compliance with the specifications of the German Council for Resuscitation. The study sample included adults with nontraumatic out-of-hospital cardiac arrest admitted between 2013 and 2018. Exposure was the PROgnostication using LOGistic regression model for Unselected adult cardiac arrest patients in the Early stages (PROLOGUE) score, including 12 clinical variables readily available at hospital admission. The outcome was poor 30-day neurologic function, as assessed using the cerebral performance category scale. The risk of a poor outcome was calculated using the PROLOGUE score regression equation. Predicted risk deciles were compared to observed outcome estimates in a complete-case analysis, a best-case analysis, and a multiple-data-imputation analysis using the Markov chain Monte Carlo method.
Results: A total of 1051 patients (median 61 years, IQR 50-71; 29% female) were analyzed. A total of 808 patients (77%) were included in the complete-case analysis. The PROLOGUE score overestimated the risk of poor neurologic outcomes in the range of 40% to 100% predicted risk, involving 63% of patients. The model fit did not improve after missing data imputation.
Conclusions: In a plausibly related sample of European cardiac arrest survivors, risk prediction by the PROLOGUE score was largely too pessimistic and failed to replicate the high accuracy found in the original study. Using the PROLOGUE score as an example, this study highlights the compelling need for independent validation of a proposed prognostic score to prevent potentially fatal mispredictions.
Keywords: calibration; cardiac arrest; data imputation; discrimination; external validation; independent sample; prognostic score; risk estimation.