Development and validation of the predictive risk of death model for adult patients admitted to intensive care units in Japan: an approach to improve the accuracy of healthcare quality measures

Hideki Endo; Shigehiko Uchino; Satoru Hashimoto; Yoshitaka Aoki; Eiji Hashiba; Junji Hatakeyama; Katsura Hayakawa; Nao Ichihara; Hiromasa Irie; Tatsuya Kawasaki; Junji Kumasawa; Hiroshi Kurosawa; Tomoyuki Nakamura; Hiroyuki Ohbe; Hiroshi Okamoto; Hidenobu Shigemitsu; Takashi Tagami; Shunsuke Takaki; Kohei Takimoto; Masatoshi Uchida; Hiroaki Miyata

doi:10.1186/s40560-021-00533-z

Development and validation of the predictive risk of death model for adult patients admitted to intensive care units in Japan: an approach to improve the accuracy of healthcare quality measures

J Intensive Care. 2021 Feb 15;9(1):18. doi: 10.1186/s40560-021-00533-z.

Authors

Hideki Endo^{1

2}, Shigehiko Uchino³, Satoru Hashimoto⁴, Yoshitaka Aoki⁵, Eiji Hashiba⁶, Junji Hatakeyama⁷, Katsura Hayakawa⁸, Nao Ichihara⁹, Hiromasa Irie¹⁰, Tatsuya Kawasaki¹¹, Junji Kumasawa¹², Hiroshi Kurosawa¹³, Tomoyuki Nakamura¹⁴, Hiroyuki Ohbe¹⁵, Hiroshi Okamoto¹⁶, Hidenobu Shigemitsu¹⁷, Takashi Tagami¹⁸, Shunsuke Takaki¹⁹, Kohei Takimoto²⁰, Masatoshi Uchida²¹, Hiroaki Miyata^{9

22}

Affiliations

¹ Department of Healthcare Quality Assessment, Graduate School of Medicine, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8655, Japan. hidendo-thk@umin.ac.jp.
² Department of Health Policy and Management, School of Medicine, Keio University, 35 Shinanomachi, Shinjuku-ku, Tokyo, 160-8582, Japan. hidendo-thk@umin.ac.jp.
³ Intensive Care Unit, The Jikei University School of Medicine, 3-19-18 Nishi-Shinbashi, Minato-ku, Tokyo, 105-8471, Japan.
⁴ Department of Anesthesiology and Intensive Care Medicine, Kyoto Prefectural University of Medicine, 465 Kajii-cho, Kawaramachi-Hirokoji, Kamigyo-ku, Kyoto, 602-8566, Japan.
⁵ Department of Anesthesiology and Intensive Care Medicine, Hamamatsu University School of Medicine, 1-20-1 Handayama, Higashi-ku, Hamamatsu, Shizuoka, 431-3192, Japan.
⁶ Division of Intensive Care, Hirosaki University Hospital, 53 Honcho, Hirosaki, Aomori, 036-8203, Japan.
⁷ Department of Emergency and Critical Care Medicine, National Hospital Organization Tokyo Medical Center, 2-5-1, Higashigaoka, Meguro-ku, Tokyo, 152-8902, Japan.
⁸ Department of Emergency and Critical Care Medicine, Saitama Red Cross Hospital, 1-5 Shintoshin, Chuo-ku, Saitama, 330-8553, Japan.
⁹ Department of Healthcare Quality Assessment, Graduate School of Medicine, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8655, Japan.
¹⁰ Department of Anesthesiology, Kurashiki Central Hospital, 1-1-1 Miwa, Kurashiki, Okayama, 710-8602, Japan.
¹¹ Department of Pediatric Critical Care, Shizuoka Children's Hospital, 860 Urushiyama, Aoi-ku, Shizuoka, Shizuoka, 420-8660, Japan.
¹² Department of Critical Care Medicine, Sakai City Medical Center, 1-1-1 Ebaraji-cho, Nishi-ku, Sakai, Osaka, 593-8304, Japan.
¹³ Department of Pediatric Critical Care Medicine, Hyogo Prefectural Kobe Children's Hospital, 1-6-7 Minatojima Minamimachi, Chuo-ku, Kobe, Hyogo, 650-0047, Japan.
¹⁴ Department of Anesthesiology and Critical Care Medicine, Fujita Health University School of Medicine, 1-98 Dengakugakubo, Kutsukake-cho, Toyoake, Aichi, 470-1192, Japan.
¹⁵ Department of Clinical Epidemiology and Health Economics, School of Public Health, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-0033, Japan.
¹⁶ Department of Critical Care Medicine, St. Luke's International Hospital, 9-1 Akashi-cho, Chuo-ku, Tokyo, 104-8560, Japan.
¹⁷ Department of Intensive Care Medicine, Graduate School of Medicine, Tokyo Medical and Dental University, 1-5-45 Yushima, Bunkyo-ku, Tokyo, 113-8519, Japan.
¹⁸ Department of Emergency and Critical Care Medicine, Nippon Medical School Musashikosugi Hospital, 1-396 Kosugimachi, Nakahara-ku, Kawasaki, Kanagawa, 211-8533, Japan.
¹⁹ Department of Anesthesiology and Critical Care Medicine, Yokohama City University, 3-9 Fukuura, Kanazawa-ku, Yokohama, Kanagawa, 236-0004, Japan.
²⁰ Department of Intensive Care Medicine, Kameda Medical Center, 929 Higashi-cho, Kamogawa, Chiba, 296-8602, Japan.
²¹ Department of Emergency and Critical Care Medicine, Dokkyo Medical University, 880 Kitakobayashi, Mibu-machi, Shimotsuga-gun, Tochigi, 321-0293, Japan.
²² Department of Health Policy and Management, School of Medicine, Keio University, 35 Shinanomachi, Shinjuku-ku, Tokyo, 160-8582, Japan.

Abstract

Background: The Acute Physiology and Chronic Health Evaluation (APACHE) III-j model is widely used to predict mortality in Japanese intensive care units (ICUs). Although the model's discrimination is excellent, its calibration is poor. APACHE III-j overestimates the risk of death, making its evaluation of healthcare quality inaccurate. This study aimed to improve the calibration of the model and develop a Japan Risk of Death (JROD) model for benchmarking purposes.

Methods: A retrospective analysis was conducted using a national clinical registry of ICU patients in Japan. Adult patients admitted to an ICU between April 1, 2018, and March 31, 2019, were included. The APACHE III-j model was recalibrated with the following models: Model 1, predicting mortality with an offset variable for the linear predictor of the APACHE III-j model using a generalized linear model; model 2, predicting mortality with the linear predictor of the APACHE III-j model using a generalized linear model; and model 3, predicting mortality with the linear predictor of the APACHE III-j model using a hierarchical generalized additive model. Model performance was assessed with the area under the receiver operating characteristic curve (AUROC), the Brier score, and the modified Hosmer-Lemeshow test. To confirm model applicability to evaluating quality of care, funnel plots of the standardized mortality ratio and exponentially weighted moving average (EWMA) charts for mortality were drawn.

Results: In total, 33,557 patients from 44 ICUs were included in the study population. ICU mortality was 3.8%, and hospital mortality was 8.1%. The AUROC, Brier score, and modified Hosmer-Lemeshow p value of the original model and models 1, 2, and 3 were 0.915, 0.062, and < .001; 0.915, 0.047, and < .001; 0.915, 0.047, and .002; and 0.917, 0.047, and .84, respectively. Except for model 3, the funnel plots showed overdispersion. The validity of the EWMA charts for the recalibrated models was determined by visual inspection.

Conclusions: Model 3 showed good performance and can be adopted as the JROD model for monitoring quality of care in an ICU, although further investigation of the clinical validity of outlier detection is required. This update method may also be useful in other settings.

Keywords: Benchmarking; Quality improvement; Quality indicator; Recalibration; Risk of death; Risk prediction model.