Robust multimodal fusion network using adversarial learning for brain tumor grading

Abstract

Background and objective: Gliomas are graded using multimodal magnetic resonance imaging, which provides important information for treatment and prognosis. When modalities are missing, the grading is degraded. We propose a robust brain tumor grading model that can handle missing modalities.

Methods: Our method was developed and tested on Brain Tumor Segmentation Challenge 2017 dataset (n = 285) via nested five-fold cross-validation. Our method adopts adversarial learning to generate the features of missing modalities relative to the features obtained from a full set of modalities in the latent space. An attention-based fusion block across modalities fuses the features of each available modality into a shared representation. Our method's results are compared to those of two other models where 15 missing-modality scenarios are explicitly considered and a joint training approach with random dropouts is used.

Results: Our method outperforms the two competing methods in classifying high-grade gliomas (HGGs) and low-grade gliomas (LGGs), achieving an area under the curve of 87.76% on average for all missing-modality scenarios. The activation maps derived with our method confirm that it focuses on the enhancing portion of the tumor in HGGs and on the edema and non-enhancing portions of the tumor in LGGs, which is consistent with prior expertise. An ablation study shows the added benefits of a fusion block and adversarial learning for handling missing modalities.

Conclusion: Our method shows robust grading of gliomas in all cases of missing modalities. Our proposed network might have positive implications in glioma care by learning features robust to missing modalities.

Keywords: Adversarial learning; Fusion block; Glioma grading; Missing modalities; Robust classification.