Congenital Heart Surgery Machine Learning-Derived In-Depth Benchmarking Tool

George E Sarris; Daisy Zhuo; Luca Mingardi; Jack Dunn; Jordan Levine; Zdzislaw Tobota; Bohdan Maruszewski; Jose Fragata; Dimitris Bertsimas

doi:10.1016/j.athoracsur.2023.10.034

Congenital Heart Surgery Machine Learning-Derived In-Depth Benchmarking Tool

Ann Thorac Surg. 2023 Dec 6:S0003-4975(23)01242-0. doi: 10.1016/j.athoracsur.2023.10.034. Online ahead of print.

Authors

George E Sarris¹, Daisy Zhuo², Luca Mingardi², Jack Dunn², Jordan Levine³, Zdzislaw Tobota⁴, Bohdan Maruszewski⁴, Jose Fragata⁵, Dimitris Bertsimas⁶

Affiliations

¹ Athens Heart Surgery Institute, Athens, Greece. Electronic address: gsarris@mac.com.
² Alexandria Health, Cambridge, Massachusetts.
³ Alexandria Health, Providence, Rhode Island.
⁴ The Children's Memorial Health Institute, Warsaw, Poland.
⁵ Department of Cardiothoracic Surgery, Hospital de Santa Marta and NOVA University, Lisbon, Portugal.
⁶ Operations Research Center and Sloan School of Management, Massachusetts Institute of Technology, Cambridge, Massachusetts.

PMID: 38065331
DOI: 10.1016/j.athoracsur.2023.10.034

Abstract

Background: We previously showed that machine learning-based methodologies of optimal classification trees (OCTs) can accurately predict risk after congenital heart surgery and assess case-mix-adjusted performance after benchmark procedures. We extend this methodology to provide interpretable, easily accessible, and actionable hospital performance analysis across all procedures.

Methods: The European Congenital Heart Surgeons Association Congenital Cardiac Database data subset of 172,888 congenital cardiac surgical procedures performed in European centers between 1989 and 2022 was analyzed. OCT models (decision trees) were built predicting hospital mortality (area under the curve [AUC], 0.866), prolonged postoperative mechanical ventilatory support time (AUC, 0.851), or hospital length of stay (AUC, 0.818), thereby establishing case-adjusted benchmarking standards reflecting the overall performance of all participating hospitals, designated as the "virtual hospital." OCT analysis of virtual hospital aggregate data yielded predicted expected outcomes (both aggregate and for risk-matched patient cohorts) for the individual hospital's own specific case-mix, readily available on-line.

Results: Raw average rates were hospital mortality, 4.9%; mechanical ventilatory support time, 14.5%; and length of stay, 15.0%. Of 146 participating centers, compared with each hospital's overall case-adjusted predicted hospital mortality benchmark, 20.5% statistically (<90% CI) overperformed and 20.5% underperformed. An interactive tool based on the OCT analysis automatically reveals 14 hospital-specific patient cohorts, simultaneously assessing overperformance or underperformance, and enabling further analysis of cohort strata in any chosen time frame.

Conclusions: Machine learning-based OCT benchmarking analysis provides automatic assessment of hospital-specific case-adjusted performance after congenital heart surgery, not only overall but importantly, also by similar risk patient cohorts. This is a tool for hospital self-assessment, particularly facilitated by the user-accessible online-platform.