Deep learning assessment compared to radiologist reporting for metastatic spinal cord compression on CT

James Thomas Patrick Decourcy Hallinan; Lei Zhu; Wenqiao Zhang; Shuliang Ge; Faimee Erwan Muhamat Nor; Han Yang Ong; Sterling Ellis Eide; Amanda J L Cheng; Tricia Kuah; Desmond Shi Wei Lim; Xi Zhen Low; Kuan Yuen Yeong; Mona I AlMuhaish; Ahmed Mohamed Alsooreti; Nesaretnam Barr Kumarakulasinghe; Ee Chin Teo; Qai Ven Yap; Yiong Huak Chan; Shuxun Lin; Jiong Hao Tan; Naresh Kumar; Balamurugan A Vellayappan; Beng Chin Ooi; Swee Tian Quek; Andrew Makmur

doi:10.3389/fonc.2023.1151073

Deep learning assessment compared to radiologist reporting for metastatic spinal cord compression on CT

Front Oncol. 2023 May 4:13:1151073. doi: 10.3389/fonc.2023.1151073. eCollection 2023.

Authors

James Thomas Patrick Decourcy Hallinan^{1

2}, Lei Zhu³, Wenqiao Zhang³, Shuliang Ge¹, Faimee Erwan Muhamat Nor^{1

2}, Han Yang Ong^{1

2}, Sterling Ellis Eide^{1

2}, Amanda J L Cheng^{1

2}, Tricia Kuah¹, Desmond Shi Wei Lim¹, Xi Zhen Low¹, Kuan Yuen Yeong⁴, Mona I AlMuhaish^{1

5}, Ahmed Mohamed Alsooreti^{1

6}, Nesaretnam Barr Kumarakulasinghe⁷, Ee Chin Teo¹, Qai Ven Yap⁸, Yiong Huak Chan⁸, Shuxun Lin⁹, Jiong Hao Tan¹⁰, Naresh Kumar¹⁰, Balamurugan A Vellayappan¹¹, Beng Chin Ooi³, Swee Tian Quek^{1

2}, Andrew Makmur^{1

2}

Affiliations

¹ Department of Diagnostic Imaging, National University Hospital, Singapore, Singapore.
² Department of Diagnostic Radiology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore.
³ Department of Computer Science, School of Computing, National University of Singapore, Singapore, Singapore.
⁴ Department of Radiology, Ng Teng Fong General Hospital, Singapore, Singapore.
⁵ Department of Radiology, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia.
⁶ Department of Diagnostic Imaging, Salmaniya Medical Complex, Manama, Bahrain.
⁷ National University Cancer Institute, National University Hospital, Singapore, Singapore.
⁸ Biostatistics Unit, Yong Loo Lin School of Medicine, Singapore, Singapore.
⁹ Division of Spine Surgery, Department of Orthopaedic Surgery, Ng Teng Fong General Hospital, Singapore, Singapore.
¹⁰ University Spine Centre, Department of Orthopaedic Surgery, National University Health System, Singapore, Singapore.
¹¹ Department of Radiation Oncology, National University Cancer Institute Singapore, National University Hospital, Singapore, Singapore.

Abstract

Introduction: Metastatic spinal cord compression (MSCC) is a disastrous complication of advanced malignancy. A deep learning (DL) algorithm for MSCC classification on CT could expedite timely diagnosis. In this study, we externally test a DL algorithm for MSCC classification on CT and compare with radiologist assessment.

Methods: Retrospective collection of CT and corresponding MRI from patients with suspected MSCC was conducted from September 2007 to September 2020. Exclusion criteria were scans with instrumentation, no intravenous contrast, motion artefacts and non-thoracic coverage. Internal CT dataset split was 84% for training/validation and 16% for testing. An external test set was also utilised. Internal training/validation sets were labelled by radiologists with spine imaging specialization (6 and 11-years post-board certification) and were used to further develop a DL algorithm for MSCC classification. The spine imaging specialist (11-years expertise) labelled the test sets (reference standard). For evaluation of DL algorithm performance, internal and external test data were independently reviewed by four radiologists: two spine specialists (Rad1 and Rad2, 7 and 5-years post-board certification, respectively) and two oncological imaging specialists (Rad3 and Rad4, 3 and 5-years post-board certification, respectively). DL model performance was also compared against the CT report issued by the radiologist in a real clinical setting. Inter-rater agreement (Gwet's kappa) and sensitivity/specificity/AUCs were calculated.

Results: Overall, 420 CT scans were evaluated (225 patients, mean age=60 ± 11.9[SD]); 354(84%) CTs for training/validation and 66(16%) CTs for internal testing. The DL algorithm showed high inter-rater agreement for three-class MSCC grading with kappas of 0.872 (p<0.001) and 0.844 (p<0.001) on internal and external testing, respectively. On internal testing DL algorithm inter-rater agreement (κ=0.872) was superior to Rad 2 (κ=0.795) and Rad 3 (κ=0.724) (both p<0.001). DL algorithm kappa of 0.844 on external testing was superior to Rad 3 (κ=0.721) (p<0.001). CT report classification of high-grade MSCC disease was poor with only slight inter-rater agreement (κ=0.027) and low sensitivity (44.0), relative to the DL algorithm with almost-perfect inter-rater agreement (κ=0.813) and high sensitivity (94.0) (p<0.001).

Conclusion: Deep learning algorithm for metastatic spinal cord compression on CT showed superior performance to the CT report issued by experienced radiologists and could aid earlier diagnosis.

Keywords: CT; Epidural spinal cord compression; MRI; artificial intelligence; deep learning; metastatic epidural spinal cord compression (MESCC); metastatic spinal cord compression (MSCC); spinal metastatic disease.

Copyright © 2023 Hallinan, Zhu, Zhang, Ge, Muhamat Nor, Ong, Eide, Cheng, Kuah, Lim, Low, Yeong, AlMuhaish, Alsooreti, Kumarakulasinghe, Teo, Yap, Chan, Lin, Tan, Kumar, Vellayappan, Ooi, Quek and Makmur.

Grants and funding

This research was funded by: 1) Direct Funding from MOH/NMRC: This research is supported by the Singapore Ministry of Health National Medical Research Council under the NMRC Clinician-scientist individual research grant, new investigator grant (CS-IRG NIG); Grant Title: Deep learning pathway for the management of spine metastases (CNIG20nov-0011, MOH-000725). 2) NCIS Centre Grant Seed Funding Program (December 2020 Grant Call); Grant Title: Artificial Intelligence for the management of vertebral metastases.