MFCNet: A multi-modal fusion and calibration networks for 3D pancreas tumor segmentation on PET-CT images

Fei Wang; Chao Cheng; Weiwei Cao; Zhongyi Wu; Heng Wang; Wenting Wei; Zhuangzhi Yan; Zhaobang Liu

doi:10.1016/j.compbiomed.2023.106657

MFCNet: A multi-modal fusion and calibration networks for 3D pancreas tumor segmentation on PET-CT images

Comput Biol Med. 2023 Mar:155:106657. doi: 10.1016/j.compbiomed.2023.106657. Epub 2023 Feb 10.

Authors

Fei Wang¹, Chao Cheng², Weiwei Cao³, Zhongyi Wu³, Heng Wang⁴, Wenting Wei⁴, Zhuangzhi Yan⁵, Zhaobang Liu⁶

Affiliations

¹ Institute of Biomedical Engineering, School of Communication and Information Engineering, Shanghai University, Shanghai, 200444, China; Department of Medical Imaging, Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, 215163, China.
² Department of Nuclear Medicine, The First Affiliated Hospital of Naval Medical University(Changhai Hospital), Shanghai, 200433, China.
³ Department of Medical Imaging, Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, 215163, China.
⁴ School of Electronic and Information Engineering, Changchun University of Science and Technology, Changchun, 130022, China.
⁵ Institute of Biomedical Engineering, School of Communication and Information Engineering, Shanghai University, Shanghai, 200444, China. Electronic address: zzyan@shu.edu.cn.
⁶ Department of Medical Imaging, Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, 215163, China. Electronic address: liuzb@sibet.ac.cn.

PMID: 36791551
DOI: 10.1016/j.compbiomed.2023.106657

Abstract

In clinical diagnosis, positron emission tomography and computed tomography (PET-CT) images containing complementary information are fused. Tumor segmentation based on multi-modal PET-CT images is an important part of clinical diagnosis and treatment. However, the existing current PET-CT tumor segmentation methods mainly focus on positron emission tomography (PET) and computed tomography (CT) feature fusion, which weakens the specificity of the modality. In addition, the information interaction between different modal images is usually completed by simple addition or concatenation operations, but this has the disadvantage of introducing irrelevant information during the multi-modal semantic feature fusion, so effective features cannot be highlighted. To overcome this problem, this paper propose a novel Multi-modal Fusion and Calibration Networks (MFCNet) for tumor segmentation based on three-dimensional PET-CT images. First, a Multi-modal Fusion Down-sampling Block (MFDB) with a residual structure is developed. The proposed MFDB can fuse complementary features of multi-modal images while retaining the unique features of different modal images. Second, a Multi-modal Mutual Calibration Block (MMCB) based on the inception structure is designed. The MMCB can guide the network to focus on a tumor region by combining different branch decoding features using the attention mechanism and extracting multi-scale pathological features using a convolution kernel of different sizes. The proposed MFCNet is verified on both the public dataset (Head and Neck cancer) and the in-house dataset (pancreas cancer). The experimental results indicate that on the public and in-house datasets, the average Dice values of the proposed multi-modal segmentation network are 74.14% and 76.20%, while the average Hausdorff distances are 6.41 and 6.84, respectively. In addition, the experimental results show that the proposed MFCNet outperforms the state-of-the-art methods on the two datasets.

Keywords: 3D convolutional neural network; Multi-modal tumor segmentation; PET-CT; Pancreas.

Publication types

Research Support, Non-U.S. Gov't

MeSH terms

Calibration
Humans
Image Processing, Computer-Assisted / methods
Imaging, Three-Dimensional / methods
Pancreatic Neoplasms*
Positron Emission Tomography Computed Tomography* / methods
Tomography, X-Ray Computed / methods