Using alert dwell time to filter universal clinical alerts: A machine learning approach

Shuo-Chen Chien; Hsuan-Chia Yang; Chun-You Chen; Chia-Hui Chien; Chun-Kung Hsu; Po-Han Chien; Yu-Chuan Jack Li

doi:10.1016/j.cmpb.2023.107696

Using alert dwell time to filter universal clinical alerts: A machine learning approach

Comput Methods Programs Biomed. 2023 Oct:240:107696. doi: 10.1016/j.cmpb.2023.107696. Epub 2023 Jun 25.

Authors

Shuo-Chen Chien¹, Hsuan-Chia Yang², Chun-You Chen³, Chia-Hui Chien⁴, Chun-Kung Hsu⁵, Po-Han Chien⁶, Yu-Chuan Jack Li⁷

Affiliations

¹ Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, Taipei 110, Taiwan; Artificial Intelligence Research and Development Center, Wan Fang Hospital, Taipei Medical University, Taipei 110, Taiwan; International Center for Health Information and Technology, College of Medical science and Technology, Taipei Medical University, Taipei 110, Taiwan.
² Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, Taipei 110, Taiwan; International Center for Health Information and Technology, College of Medical science and Technology, Taipei Medical University, Taipei 110, Taiwan; Research Center of Big Data and Meta-analysis, Wan Fang Hospital, Taipei Medical University, Taipei 110, Taiwan.
³ Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, Taipei 110, Taiwan; Artificial Intelligence Research and Development Center, Wan Fang Hospital, Taipei Medical University, Taipei 110, Taiwan; International Center for Health Information and Technology, College of Medical science and Technology, Taipei Medical University, Taipei 110, Taiwan; Department of Radiation Oncology, Wan Fang Hospital, Taipei Medical University, Taipei 110, Taiwan.
⁴ Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, Taipei 110, Taiwan; International Center for Health Information and Technology, College of Medical science and Technology, Taipei Medical University, Taipei 110, Taiwan; Office of Public Affairs, Taipei Medical University, Taipei 110, Taiwan.
⁵ Office of Information Technology, Taipei Medical University, Taipei 110, Taiwan.
⁶ Department of Finance, National Taiwan University, Taipei 110, Taiwan.
⁷ Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, Taipei 110, Taiwan; International Center for Health Information and Technology, College of Medical science and Technology, Taipei Medical University, Taipei 110, Taiwan; Department of Dermatology, Wan Fang Hospital, Taipei Medical University, Taipei 110, Taiwan; Research Center for Artificial Intelligence in Medicine, Taipei Medical University, Taipei 110, Taiwan. Electronic address: jack@tmu.edu.tw.

PMID: 37480643
DOI: 10.1016/j.cmpb.2023.107696

Abstract

Background: Alerts in computerized physician order entry (CPOE) systems can improve patient safety. However, alerts in rule-based systems cannot be customized based on individual patient or user characteristics. This limitation can lead to the presentation of irrelevant alerts and subsequent alert fatigue.

Objective: We used machine learning approaches with alert dwell time to filter out irrelevant alerts for physicians based on contextual factors.

Methods: We utilized five machine learning algorithms and a total of 1,120 features grouped into six categories: alert, demographic, environment, diagnosis, prescription, and laboratory results. The output of the models was the alert dwell time within a specified time window to determine the optimal range by the sensitivity analysis.

Results: We used 813,026 records (19 categories) from the hospital's outpatient clinic data from 2020 to 2021. The sensitivity analysis showed that a time window with a range of 0.3-4.0 s had the best performance, with an area under the receiver operating characteristic (AUROC) curve of 0.73 and an area under the precision-recall curve (AUPRC) of 0.97. The model built with alert and demographic feature groups showed the best performance, with an AUROC of 0.73. The most significant individual feature groups were alert and demographic, with AUROCs of 0.66 and 0.62, respectively.

Conclusion: Our study found that alerts and user and patient demographic features are more crucial than clinical features when constructing universal context-aware alerts. Using alert dwell time in combination with a time window is an effective way to determine the trigger status of an alert. The findings of this study can provide useful insights for researchers working on specific and universal context-aware alerts.

Keywords: Alert dwell time; Alert fatigue; Context-aware alert; Interruptive alert; Machine learning.

MeSH terms

Algorithms*
Area Under Curve
Awareness*
Humans
Machine Learning
Patient Safety