Temporal validation of 30-day mortality prediction models for transcatheter aortic valve implantation using statistical process control - An observational study in a national population

Ricardo R Lopes; Tsvetan T R Yordanov; Anita A C J Ravelli; Saskia Houterman; Marije Vis; Bas A J M de Mol; Henk Marquering; Ameen Abu-Hanna; THI Registration Committee of the Netherlands Heart Registration

doi:10.1016/j.heliyon.2023.e17139

Temporal validation of 30-day mortality prediction models for transcatheter aortic valve implantation using statistical process control - An observational study in a national population

Heliyon. 2023 Jun 10;9(6):e17139. doi: 10.1016/j.heliyon.2023.e17139. eCollection 2023 Jun.

Authors

Ricardo R Lopes^{1

2

3}, Tsvetan T R Yordanov^{4

5}, Anita A C J Ravelli^{4

5}, Saskia Houterman⁶, Marije Vis^{7

8}, Bas A J M de Mol^{3

9}, Henk Marquering^{1

2

8}, Ameen Abu-Hanna^{4

5}; THI Registration Committee of the Netherlands Heart Registration

Affiliations

¹ Amsterdam UMC Location University of Amsterdam, Biomedical Engineering and Physics, Meibergdreef 9, Amsterdam, the Netherlands.
² Amsterdam UMC Location University of Amsterdam, Radiology and Nuclear Medicine, Meibergdreef 9, Amsterdam, the Netherlands.
³ Amsterdam Cardiovascular Sciences, Heart Failure and Arrhythmias, Amsterdam, the Netherlands.
⁴ Amsterdam UMC Location University of Amsterdam, Medical Informatics, Meibergdreef 9, Amsterdam, the Netherlands.
⁵ Amsterdam Public Health Research Institute, Amsterdam, the Netherlands.
⁶ Netherlands Heart Registration, Utrecht, the Netherlands.
⁷ Amsterdam UMC Location University of Amsterdam, Cardiology, Meibergdreef 9, Amsterdam, the Netherlands.
⁸ Amsterdam Cardiovascular Sciences, Atherosclerosis and Ischemic Syndromes, Amsterdam, the Netherlands.
⁹ Amsterdam UMC Location University of Amsterdam, Cardiothoracic Surgery, Meibergdreef 9, Amsterdam, the Netherlands.

Abstract

Background: Various mortality prediction models for Transcatheter Aortic Valve Implantation (TAVI) have been developed in the past years. The effect of time on the performance of such models, however, is unclear given the improvements in the procedure and changes in patient selection, potentially jeopardizing the usefulness of the prediction models in clinical practice. We aim to explore how time affects the performance and stability of different types of prediction models of 30-day mortality after TAVI.

Methods: We developed both parametric (Logistic Regression) and non-parametric (XGBoost) models to predict 30-day mortality after TAVI using data from the Netherlands Heart Registration. The models were trained with data from 2013 to the beginning of 2016 and pre-control charts from Statistical Process Control were used to analyse how time affects the models' performance on independent data from the mid of 2016 to the end of 2019. The area under the Receiver Operating Characteristics curve (AUC) was used to evaluate the models in terms of discrimination and the Brier Score (BS), which is related to calibration, in terms of accuracy of the predicted probabilities. To understand the extent to which refitting the models contribute to the models' stability, we also allowed the models to be updated over time.

Results: We included data from 11,291 consecutive TAVI patients from hospitals in the Netherlands. The parametric model without re-training had a median AUC of 0.64 (IQR 0.54-0.73) and BS of 0.028 (IQR 0.021-0.035). For the non-parametric model, the median AUC was 0.63 (IQR 0.48-0.68) and BS was 0.027 (IQR 0.021-0.036). Over time, the developed parametric model was stable in terms of AUC and unstable in terms of BS. The non-parametric model was considered unstable in both AUC and BS. Repeated model refitting resulted in stable models in terms of AUC and decreased the variability of BS, although BS was still unstable. The refitted parametric model had a median AUC of 0.66 (IQR 0.57-0.73) and BS of 0.027 (IQR 0.020-0.035) while the non-parametric model had a median AUC of 0.66 (IQR 0.57-0.74) and BS of 0.027 (IQR 0.023-0.035).

Conclusions: The temporal validation of the TAVI 30-day mortality prediction models showed that the models refitted over time are more stable and accurate when compared to the frozen models. This highlights the importance of repeatedly refitted models over time to improve or at least maintain their performance stability. The non-parametric approach did not show improvement over the parametric approach.

Keywords: Aortic stenosis; Machine learning; Prediction models; Statistical process control; Temporal validation; Transcatheter aortic valve implantation.