Predicting Postoperative Mortality With Deep Neural Networks and Natural Language Processing: Model Development and Validation

Pei-Fu Chen; Lichin Chen; Yow-Kuan Lin; Guo-Hung Li; Feipei Lai; Cheng-Wei Lu; Chi-Yu Yang; Kuan-Chih Chen; Tzu-Yu Lin

doi:10.2196/38241

Predicting Postoperative Mortality With Deep Neural Networks and Natural Language Processing: Model Development and Validation

JMIR Med Inform. 2022 May 10;10(5):e38241. doi: 10.2196/38241.

Authors

Pei-Fu Chen^#^{1

2}, Lichin Chen^#³, Yow-Kuan Lin^{1

4}, Guo-Hung Li¹, Feipei Lai^{1

5

6}, Cheng-Wei Lu^{2

7}, Chi-Yu Yang^{8

9}, Kuan-Chih Chen^{1

10}, Tzu-Yu Lin^{2

7}

Affiliations

¹ Graduate Institute of Biomedical Electronics and Bioinformatics, National Taiwan University, Taipei, Taiwan.
² Department of Anesthesiology, Far Eastern Memorial Hospital, New Taipei City, Taiwan.
³ Research Center for Information Technology Innovation, Academia Sinica, Taipei, Taiwan.
⁴ Department of Computer Science, Columbia University, New York, NY, United States.
⁵ Department of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan.
⁶ Department of Electrical Engineering, National Taiwan University, Taipei, Taiwan.
⁷ Department of Mechanical Engineering, Yuan Ze University, Taoyuan, Taiwan.
⁸ Department of Information Technology, Far Eastern Memorial Hospital, New Taipei City, Taiwan.
⁹ Section of Cardiovascular Medicine, Cardiovascular Center, Far Eastern Memorial Hospital, New Taipei City, Taiwan.
¹⁰ Department of Internal Medicine, Far Eastern Memorial Hospital, New Taipei City, Taiwan.

^# Contributed equally.

PMID: 35536634
PMCID: PMC9131148
DOI: 10.2196/38241

Abstract

Background: Machine learning (ML) achieves better predictions of postoperative mortality than previous prediction tools. Free-text descriptions of the preoperative diagnosis and the planned procedure are available preoperatively. Because reading these descriptions helps anesthesiologists evaluate the risk of the surgery, we hypothesized that deep learning (DL) models with unstructured text could improve postoperative mortality prediction. However, it is challenging to extract meaningful concept embeddings from this unstructured clinical text.

Objective: This study aims to develop a fusion DL model containing structured and unstructured features to predict the in-hospital 30-day postoperative mortality before surgery. ML models for predicting postoperative mortality using preoperative data with or without free clinical text were assessed.

Methods: We retrospectively collected preoperative anesthesia assessments, surgical information, and discharge summaries of patients undergoing general and neuraxial anesthesia from electronic health records (EHRs) from 2016 to 2020. We first compared the deep neural network (DNN) with other models using the same input features to demonstrate effectiveness. Then, we combined the DNN model with bidirectional encoder representations from transformers (BERT) to extract information from clinical texts. The effects of adding text information on the model performance were compared using the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC). Statistical significance was evaluated using P<.05.

Results: The final cohort contained 121,313 patients who underwent surgeries. A total of 1562 (1.29%) patients died within 30 days of surgery. Our BERT-DNN model achieved the highest AUROC (0.964, 95% CI 0.961-0.967) and AUPRC (0.336, 95% CI 0.276-0.402). The AUROC of the BERT-DNN was significantly higher compared to logistic regression (AUROC=0.952, 95% CI 0.949-0.955) and the American Society of Anesthesiologist Physical Status (ASAPS AUROC=0.892, 95% CI 0.887-0.896) but not significantly higher compared to the DNN (AUROC=0.959, 95% CI 0.956-0.962) and the random forest (AUROC=0.961, 95% CI 0.958-0.964). The AUPRC of the BERT-DNN was significantly higher compared to the DNN (AUPRC=0.319, 95% CI 0.260-0.384), the random forest (AUPRC=0.296, 95% CI 0.239-0.360), logistic regression (AUPRC=0.276, 95% CI 0.220-0.339), and the ASAPS (AUPRC=0.149, 95% CI 0.107-0.203).

Conclusions: Our BERT-DNN model has an AUPRC significantly higher compared to previously proposed models using no text and an AUROC significantly higher compared to logistic regression and the ASAPS. This technique helps identify patients with higher risk from the surgical description text in EHRs.

Keywords: anesthesia; anesthesiologist; bidirectional encoder representations from transformers; deep learning model; deep neural network; electronic health record; machine learning; natural language processing; neural network; postoperative mortality prediction; prediction model; preoperative medicine; unstructured text.

©Pei-Fu Chen, Lichin Chen, Yow-Kuan Lin, Guo-Hung Li, Feipei Lai, Cheng-Wei Lu, Chi-Yu Yang, Kuan-Chih Chen, Tzu-Yu Lin. Originally published in JMIR Medical Informatics (https://medinform.jmir.org), 10.05.2022.