Background: In the era of artificial intelligence, event prediction models are abundant. However, considering the limitation of the electronic medical record-based model, including the temporally skewed prediction and the record itself, these models could be delayed or could yield errors.
Objective: In this study, we aim to develop multiple event prediction models in intensive care units to overcome their temporal skewness and evaluate their robustness against delayed and erroneous input.
Methods: A total of 21,738 patients were included in the development cohort. Three events-death, sepsis, and acute kidney injury-were predicted. To overcome the temporal skewness, we developed three models for each event, which predicted the events in advance of three prespecified timepoints. Additionally, to evaluate the robustness against input error and delays, we added simulated errors and delayed input and calculated changes in the area under the receiver operating characteristic curve (AUROC) values.
Results: Most of the AUROC and area under the precision-recall curve values of each model were higher than those of the conventional scores, as well as other machine learning models previously used. In the error input experiment, except for our proposed model, an increase in the noise added to the model lowered the resulting AUROC value. However, the delayed input did not show the performance decreased in this experiment.
Conclusions: For a prediction model that was applicable in the real world, we considered not only performance but also temporal skewness, delayed input, and input error.
Keywords: AKI; critical care; intensive care unit; machine learning; mortality; prediction model; sepsis.
©MinDong Sung, Sangchul Hahn, Chang Hoon Han, Jung Mo Lee, Jayoung Lee, Jinkyu Yoo, Jay Heo, Young Sam Kim, Kyung Soo Chung. Originally published in JMIR Medical Informatics (https://medinform.jmir.org), 04.11.2021.