Modality-based attention and dual-stream multiple instance convolutional neural network for predicting microvascular invasion of hepatocellular carcinoma

Zhi Li; Yutao Wang; Yuzhao Zhu; Jiafeng Xu; Jinzhu Wei; Jiang Xie; Jian Zhang

doi:10.3389/fonc.2023.1195110

Modality-based attention and dual-stream multiple instance convolutional neural network for predicting microvascular invasion of hepatocellular carcinoma

Front Oncol. 2023 Jun 26:13:1195110. doi: 10.3389/fonc.2023.1195110. eCollection 2023.

Authors

Zhi Li^{1

2}, Yutao Wang³, Yuzhao Zhu², Jiafeng Xu², Jinzhu Wei¹, Jiang Xie⁴, Jian Zhang²

Affiliations

¹ School of Medicine, Shanghai University, Shanghai, China.
² Shanghai Universal Medical Imaging Diagnostic Center, Shanghai University, Shanghai, China.
³ The First Affiliated Hospital of Ningbo University, Ningbo, China.
⁴ School of Computer Engineering and Science, Shanghai University, Shanghai, China.

Abstract

Background and purpose: The presence of microvascular invasion (MVI) is a crucial indicator of postoperative recurrence in patients with hepatocellular carcinoma (HCC). Detecting MVI before surgery can improve personalized surgical planning and enhance patient survival. However, existing automatic diagnosis methods for MVI have certain limitations. Some methods only analyze information from a single slice and overlook the context of the entire lesion, while others require high computational resources to process the entire tumor with a three-dimension (3D) convolutional neural network (CNN), which could be challenging to train. To address these limitations, this paper proposes a modality-based attention and dual-stream multiple instance learning(MIL) CNN.

Materials and methods: In this retrospective study, 283 patients with histologically confirmed HCC who underwent surgical resection between April 2017 and September 2019 were included. Five magnetic resonance (MR) modalities including T2-weighted, arterial phase, venous phase, delay phase and apparent diffusion coefficient images were used in image acquisition of each patient. Firstly, Each two-dimension (2D) slice of HCC magnetic resonance image (MRI) was converted into an instance embedding. Secondly, modality attention module was designed to emulates the decision-making process of doctors and helped the model to focus on the important MRI sequences. Thirdly, instance embeddings of 3D scans were aggregated into a bag embedding by a dual-stream MIL aggregator, in which the critical slices were given greater consideration. The dataset was split into a training set and a testing set in a 4:1 ratio, and model performance was evaluated using five-fold cross-validation.

Results: Using the proposed method, the prediction of MVI achieved an accuracy of 76.43% and an AUC of 74.22%, significantly surpassing the performance of the baseline methods.

Conclusion: Our modality-based attention and dual-stream MIL CNN can achieve outstanding results for MVI prediction.

Keywords: MRI; attention mechanism; hepatocellular carcinoma; microvascular invasion; multiple instance learning.

Grants and funding

This work were supported by the Shanghai Municipal Health Commission Research Project (No. 201840082 and 202040068, respectively), The National Natural Science Foundation of China (No. 61873156), the Public Welfare Science and Technology Project of Ningbo (No. 2022S043).