HyperTDP-Net: A Hyper-densely Connected Compression-and-Decomposition Network Based on Trident Dilated Perception for PET and MRI Image Fusion

Abstract

Objective. Since medical images generated by medical devices have low spatial resolution and quality, fusion approaches on medical images can generate a fused image containing a more comprehensive range of different modal features to help physicians accurately diagnose diseases. Conventional methods based on deep learning for medical image fusion usually extract only local features without considering their global features, which often leads to the problem of unclear detail information in the final fused image. Therefore, medical image fusion is a challenging task of great relevance.Approach.This paper proposes a novel end-to-end medical image fusion model for PET and MRI images to achieve information interaction between different pathways, termed as hyper-densely connected compression-and-decomposition network based on trident dilated perception for PET and MRI image fusion (HyperTDP-Net). In particular, in the compression network, a dual residual hyper densely module is constructed to take full advantage of middle layer information. Moreover, we establish the trident dilated perception module to precisely determine the location information of features, and improve the feature representation capability of the network. In addition, we abandon the ordinary mean square error as the content loss function and propose a new content-aware loss consisting of structural similarity loss and gradient loss, so that the fused image not only contains rich texture details but also maintains sufficient structural similarity with the source images.Main results. The experimental dataset used in this paper is derived from multimodal medical images published by Harvard Medical School. Extensive experiments illustrate that our model contains more edge information and texture detail information in the fusion result than the 12 state-of-the-art fusion models and ablation study results demonstrate the effectiveness of three technical innovations.Significance. As medical images continue to be used in clinical diagnosis, our method is expected to be a tool that can effectively improve the accuracy of physician diagnosis and automatic machine detection.

Keywords: PET and MRI image fusion; compression-and-decomposition; dual residual hyper densely connections; trident dilated perception.