Development and validation of a transformer-based CAD model for improving the consistency of BI-RADS category 3-5 nodule classification among radiologists: a multiple center study

Hongtao Ji; Qiang Zhu; Teng Ma; Yun Cheng; Shuai Zhou; Wei Ren; Huilian Huang; Wen He; Haitao Ran; Litao Ruan; Yanli Guo; Jiawei Tian; Wu Chen; Luzeng Chen; Zhiyuan Wang; Qi Zhou; Lijuan Niu; Wei Zhang; Ruimin Yang; Qin Chen; Ruifang Zhang; Hui Wang; Li Li; Minghui Liu; Fang Nie; Aiyun Zhou

doi:10.21037/qims-22-1091

Development and validation of a transformer-based CAD model for improving the consistency of BI-RADS category 3-5 nodule classification among radiologists: a multiple center study

Quant Imaging Med Surg. 2023 Jun 1;13(6):3671-3687. doi: 10.21037/qims-22-1091. Epub 2023 Apr 28.

Authors

Hongtao Ji¹, Qiang Zhu¹, Teng Ma¹, Yun Cheng¹, Shuai Zhou¹, Wei Ren¹, Huilian Huang¹, Wen He², Haitao Ran³, Litao Ruan⁴, Yanli Guo⁵, Jiawei Tian⁶, Wu Chen⁷, Luzeng Chen⁸, Zhiyuan Wang⁹, Qi Zhou¹⁰, Lijuan Niu¹¹, Wei Zhang¹², Ruimin Yang¹³, Qin Chen¹⁴, Ruifang Zhang¹⁵, Hui Wang¹⁶, Li Li¹⁷, Minghui Liu¹⁸, Fang Nie¹⁹, Aiyun Zhou²⁰

Affiliations

¹ Department of Diagnostic Ultrasound, Beijing Tongren Hospital, Capital Medical University, Beijing, China.
² Department of Ultrasonography, Beijing Tiantan Hospital, Capital Medical University, Beijing, China.
³ Department of Ultrasound, The Second Affiliated Hospital, Chongqing Medical University, Chongqing, China.
⁴ Department of Medical Ultrasound, The First Affiliated Hospital, Xi'an Jiaotong University, Xi'an, China.
⁵ Department of Ultrasound, The Southwest Hospital, Army Medical University, Chongqing, China.
⁶ Department of Ultrasound, The Second Affiliated Hospital, Harbin Medical University, Harbin, China.
⁷ Department of Ultrasound, The First Hospital, Shanxi Medical University, Taiyuan, China.
⁸ Department of Ultrasound, The First Hospital, Peking University, Beijing, China.
⁹ Department of Ultrasound, Diagnosis Center of Ultrasound, Hunan Province Cancer Hospital, Changsha, China.
¹⁰ Department of Ultrasound, The Second Affiliated Hospital, Xi'an Jiaotong University, Xi'an, China.
¹¹ Department of Ultrasound, Cancer Hospital, National Cancer Center, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China.
¹² Department of Ultrasonography, The Third Affiliated Hospital, Guangxi Medical University, Nanning, China.
¹³ Department of Ultrasound, The Frist Affiliated Hospital of Hebei North University, Zhangjiakou, China.
¹⁴ Department of Ultrasound, Sichuan Provincial People's Hospital, University of Electronic Science and Technology of China, Chengdu, China.
¹⁵ Department of Ultrasound, The First Affiliated Hospital, Zhengzhou University, Zhengzhou, China.
¹⁶ Department of Ultrasound, China-Japan Union Hospital, Jilin University, Changchun, China.
¹⁷ Department of Ultrasound, Qilu Hospital of Shandong University, Qingdao, China.
¹⁸ Department of Ultrasound Diagnosis, The Second Xiangya Hospital, Central South University, Changsha, China.
¹⁹ Department of Ultrasound, Lanzhou University Second Hospital, Lanzhou, China.
²⁰ Department of Ultrasound, The First Affiliated Hospital, Nanchang University, Nanchang, China.

Abstract

Background: Significant differences exist in the classification outcomes for radiologists using ultrasonography-based Breast Imaging Reporting and Data Systems for diagnosing category 3-5 (BI-RADS 3-5) breast nodules, due to a lack of clear and distinguishing image features. Consequently, this retrospective study investigated the improvement of BI-RADS 3-5 classification consistency using a transformer-based computer-aided diagnosis (CAD) model.

Methods: Independently, 5 radiologists performed BI-RADS annotations on 21,332 breast ultrasonographic images collected from 3,978 female patients from 20 clinical centers in China. All images were divided into training, validation, testing, and sampling sets. The trained transformer-based CAD model was then used to classify test images, for which sensitivity (SEN), specificity (SPE), accuracy (ACC), area under the curve (AUC), and calibration curve were evaluated. Variations in these metrics among the 5 radiologists were analyzed by referencing BI-RADS classification results for the sampling test set provided by CAD to determine whether classification consistency (the k value), SEN, SPE, and ACC could be improved.

Results: After the training set (11,238 images) and validation set (2,996 images) were learned by the CAD model, the classification ACC of the CAD model applied to the test set (7,098 images) was 94.89% in category 3, 96.90% in category 4A, 95.49% in category 4B, 92.28% in category 4C, and 95.45% in category 5 nodules. Based on pathological results, the AUC of the CAD model was 0.924 and the predicted probability of CAD was a little higher than the actual probability in the calibration curve. After referencing BI-RADS classification results, the adjustments were made to 1,583 nodules, of which 905 were classified to a lower category and 678 to a higher category in the sampling test set. As a result, the ACC (72.41-82.65%), SEN (32.73-56.98%), and SPE (82.46-89.26%) of the classification by each radiologist were significantly improved on average, with the consistency (k values) in almost all of them increasing to >0.6.

Conclusions: The radiologist's classification consistency was markedly improved with almost all the k values increasing by a value greater than 0.6, and the diagnostic efficiency was also improved by approximately 24% (32.73% to 56.98%) and 7% (82.46% to 89.26%) for SEN and SPE, respectively, of the total classification on average. The transformer-based CAD model can help to improve the radiologist's diagnostic efficacy and consistency with others in the classification of BI-RADS 3-5 nodules.

Keywords: Breast Imaging Reporting and Data Systems (BI-RADS); computer-aided diagnosis (CAD); transformers; ultrasound.