Improving Machine Learning 30-Day Mortality Prediction by Discounting Surprising Deaths

Abstract

Background: Machine learning (ML) is an emerging tool for predicting need of end-of-life discussion and palliative care, by using mortality as a proxy. But deaths, unforeseen by emergency physicians at time of the emergency department (ED) visit, might have a weaker association with the ED visit.

Objectives: To develop an ML algorithm that predicts unsurprising deaths within 30 days after ED discharge.

Methods: In this retrospective registry study, we included all ED attendances within the Swedish region of Halland in 2015 and 2016. All registered deaths within 30 days after ED discharge were classified as either "surprising" or "unsurprising" by an adjudicating committee with three senior specialists in emergency medicine. ML algorithms were developed for the death subclasses by using Logistic Regression (LR), Random Forest (RF), and Support Vector Machine (SVM).

Results: Of all 30-day deaths (n = 148), 76% (n = 113) were not surprising to the adjudicating committee. The most common diseases were advanced stage cancer, multidisease/frailty, and dementia. By using LR, RF, and SVM, mean area under the receiver operating characteristic curve (ROC-AUC) of unsurprising deaths in the test set were 0.950 (SD 0.008), 0.944 (SD 0.007), and 0.949 (SD 0.007), respectively. For all mortality, the ROC-AUCs for LR, RF, and SVM were 0.924 (SD 0.012), 0.922 (SD 0.009), and 0.931 (SD 0.008). The difference in prediction performance between all and unsurprising death was statistically significant (P < .001) for all three models.

Conclusion: In patients discharged to home from the ED, three-quarters of all 30-day deaths did not surprise an adjudicating committee with emergency medicine specialists. When only unsurprising deaths were included, ML mortality prediction improved significantly.

Keywords: artificial intelligence; emergency department; emergency medicine; end-of-life; machine learning; palliative care.