Comparison of discrimination and calibration performance of ECG-based machine learning models for prediction of new-onset atrial fibrillation

Giovanni Baj; Ilaria Gandin; Arjuna Scagnetto; Luca Bortolussi; Chiara Cappelletto; Andrea Di Lenarda; Giulia Barbati

doi:10.1186/s12874-023-01989-3

Comparison of discrimination and calibration performance of ECG-based machine learning models for prediction of new-onset atrial fibrillation

BMC Med Res Methodol. 2023 Jul 22;23(1):169. doi: 10.1186/s12874-023-01989-3.

Authors

Giovanni Baj¹, Ilaria Gandin², Arjuna Scagnetto³, Luca Bortolussi⁴, Chiara Cappelletto³, Andrea Di Lenarda³, Giulia Barbati²

Affiliations

¹ Department of Mathematics and Geosciences, University of Trieste, Trieste, Italy. giovanni.baj@phd.units.it.
² Department of Medical Sciences, Biostatistics Unit, University of Trieste, Trieste, Italy.
³ Cardiovascular Center, Territorial Specialistic Department, University Hospital and Health Services of Trieste, Trieste, Italy.
⁴ Department of Mathematics and Geosciences, University of Trieste, Trieste, Italy.

Abstract

Background: Machine learning (ML) methods to build prediction models starting from electrocardiogram (ECG) signals are an emerging research field. The aim of the present study is to investigate the performances of two ML approaches based on ECGs for the prediction of new-onset atrial fibrillation (AF), in terms of discrimination, calibration and sample size dependence.

Methods: We trained two models to predict new-onset AF: a convolutional neural network (CNN), that takes as input the raw ECG signals, and an eXtreme Gradient Boosting model (XGB), that uses the signal's extracted features. A penalized logistic regression model (LR) was used as a benchmark. Discrimination was evaluated with the area under the ROC curve, while calibration with the integrated calibration index. We investigated the dependence of models' performances on the sample size and on class imbalance corrections introduced with random under-sampling.

Results: CNN's discrimination was the most affected by the sample size, outperforming XGB and LR only around n = 10.000 observations. Calibration showed only a small dependence on the sample size for all the models considered. Balancing the training set with random undersampling did not improve discrimination in any of the models. Instead, the main effect of imbalance corrections was to worsen the models' calibration (for CNN, integrated calibration index from 0.014 [0.01, 0.018] to 0.17 [0.16, 0.19]). The sample size emerged as a fundamental point for developing the CNN model, especially in terms of discrimination (AUC = 0.75 [0.73, 0.77] when n = 10.000, AUC = 0.80 [0.79, 0.81] when n = 150.000). The effect of the sample size on the other two models was weaker. Imbalance corrections led to poorly calibrated models, for all the approaches considered, reducing the clinical utility of the models.

Conclusions: Our results suggest that the choice of approach in the analysis of ECG should be based on the amount of data available, preferring more standard models for small datasets. Moreover, imbalance correction methods should be avoided when developing clinical prediction models, where calibration is crucial.

Keywords: Atrial fibrillation; Calibration; Deep learning; Machine learning; Prediction.

MeSH terms

Atrial Fibrillation* / diagnosis
Benchmarking
Calibration
Electrocardiography
Humans
Machine Learning