Contrast between traditional and machine learning algorithms based on a urine culture predictive model: a multicenter retrospective study in patients with urinary calculi

Yuhui He; Panxin Peng; Wenwei Ying; Qinwei Wang; Yan Wang; Xiankui Liu; Wenhui Song; Yue Gao; Peizhe Li; Jie Wang; Weijie Zhu; Wenzhi Gao; Xiaofeng Zhou; Xuesong Li; Liqun Zhou

doi:10.21037/tau-21-780

Contrast between traditional and machine learning algorithms based on a urine culture predictive model: a multicenter retrospective study in patients with urinary calculi

Transl Androl Urol. 2022 Feb;11(2):139-148. doi: 10.21037/tau-21-780.

Authors

Yuhui He^#¹, Panxin Peng^#², Wenwei Ying^#¹, Qinwei Wang³, Yan Wang³, Xiankui Liu⁴, Wenhui Song⁵, Yue Gao⁵, Peizhe Li², Jie Wang¹, Weijie Zhu¹, Wenzhi Gao¹, Xiaofeng Zhou², Xuesong Li¹, Liqun Zhou¹

Affiliations

¹ Department of Urology, Peking University First Hospital, Beijing, China.
² Department of Urology, China-Japan Friendship Hospital, Beijing, China.
³ Department of Urology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, China.
⁴ Department of Urology, The First Affiliated Hospital of China Medical University, Shenyang, China.
⁵ Department of Urology, Tianjin First Central Hospital, Tianjin, China.

^# Contributed equally.

Abstract

Background: Quick and accurate identification of urinary calculi patients with positive urinary cultures is critical to the choice of the treatment strategy. Predictive models based on machine learning algorithms provide a new way to solve this problem. This study aims to determine the predictive value of machine learning algorithms using a urine culture predictive model based on patients with urinary calculi.

Methods: Data were collected from four clinical centers in the period of June 2016, to May 2019. 2,054 cases were included in the study. The dataset was randomly split into ratios of 5:5, 6:4, and 7:3 for model construction and validation. Predictive models of urine culture outcomes were constructed and validated by logistic regression, random forest, adaboost, and gradient boosting decision tree (GBDT) models. Each ratio's construction and verification were repeated five times independently for cross-validation. The Matthews correlation coefficient (MMC), F1-score, receiver operating characteristic (ROC) curve with the area under curve (AUC) was used to evaluate the performance of each prediction model. The additive net reclassification index (NRI) and absolute NRI were used to assess the predictive capabilities of the models.

Results: Four prediction models of urinary culture results in patients with urinary calculi were constructed. The mean AUCs of the logistic regression, random forest, adaboost, and GBDT models were 0.761 (95% CI: 0.753-0.770), 0.790 (95% CI: 0.782-0.798), 0.779 (95% CI: 0.766-0.791), and 0.831 (95% CI: 0.823-0.840), respectively. Moreover, the average MMC and F1-score of GBDT model was 0.460 and 0.588, which was improved compared to logistic regression model of 0.335 and 0.501. The additive NRI and absolute NRI of the GBDT and logistic regression models were 0.124 (95% CI: 0.106-0.142) and 0.065 (95% CI: 0.060-0.069), respectively.

Conclusions: Our results indicate that machine learning algorithms may be useful tools for urine culture outcome prediction in patients with urinary calculi because they exhibit superior performance compared with the logistic regression model.

Keywords: Logistic regression; bacteriuria; early diagnosis; machine learning; urinary calculi.