Attention-based Saliency Maps Improve Interpretability of Pneumothorax Classification

Alessandro Wollek; Robert Graf; Saša Čečatka; Nicola Fink; Theresa Willem; Bastian O Sabel; Tobias Lasser

doi:10.1148/ryai.220187

Attention-based Saliency Maps Improve Interpretability of Pneumothorax Classification

Radiol Artif Intell. 2022 Mar 1;5(2):e220187. doi: 10.1148/ryai.220187. eCollection 2023 Mar.

Authors

Alessandro Wollek^#¹, Robert Graf^#¹, Saša Čečatka¹, Nicola Fink¹, Theresa Willem¹, Bastian O Sabel¹, Tobias Lasser¹

Affiliation

¹ Munich Institute of Biomedical Engineering and Department of Informatics, Technical University of Munich, Boltzmannstr 11, Garching b., Munich 85748, Germany (A.W., R.G., T.L.); Department of Radiology, University Hospital LMU, Munich, Germany (S.Č., N.F., B.O.S.); and Munich School of Technology in Society, Technical University of Munich, Munich, Germany (T.W.).

^# Contributed equally.

Abstract

Purpose: To investigate the chest radiograph classification performance of vision transformers (ViTs) and interpretability of attention-based saliency maps, using the example of pneumothorax classification.

Materials and methods: In this retrospective study, ViTs were fine-tuned for lung disease classification using four public datasets: CheXpert, Chest X-Ray 14, MIMIC CXR, and VinBigData. Saliency maps were generated using transformer multimodal explainability and gradient-weighted class activation mapping (GradCAM). Classification performance was evaluated on the Chest X-Ray 14, VinBigData, and Society for Imaging Informatics in Medicine-American College of Radiology (SIIM-ACR) Pneumothorax Segmentation datasets using the area under the receiver operating characteristic curve (AUC) analysis and compared with convolutional neural networks (CNNs). The explainability methods were evaluated with positive and negative perturbation, sensitivity-n, effective heat ratio, intra-architecture repeatability, and interarchitecture reproducibility. In the user study, three radiologists classified 160 chest radiographs with and without saliency maps for pneumothorax and rated their usefulness.

Results: ViTs had comparable chest radiograph classification AUCs compared with state-of-the-art CNNs: 0.95 (95% CI: 0.94, 0.95) versus 0.83 (95%, CI 0.83, 0.84) on Chest X-Ray 14, 0.84 (95% CI: 0.77, 0.91) versus 0.83 (95% CI: 0.76, 0.90) on VinBigData, and 0.85 (95% CI: 0.85, 0.86) versus 0.87 (95% CI: 0.87, 0.88) on SIIM-ACR. Both saliency map methods unveiled a strong bias toward pneumothorax tubes in the models. Radiologists found 47% of the attention-based and 39% of the GradCAM saliency maps useful. The attention-based methods outperformed GradCAM on all metrics.

Conclusion: ViTs performed similarly to CNNs in chest radiograph classification, and their attention-based saliency maps were more useful to radiologists and outperformed GradCAM.Keywords: Conventional Radiography, Thorax, Diagnosis, Supervised Learning, Convolutional Neural Network (CNN) Online supplemental material is available for this article. © RSNA, 2023.

Keywords: Conventional Radiography; Convolutional Neural Network (CNN); Diagnosis; Supervised Learning; Thorax.