STNet: shape and texture joint learning through two-stream network for knowledge-guided image recognition

Xijing Wang; Hongcheng Han; Mengrui Xu; Shengpeng Li; Dong Zhang; Shaoyi Du; Meifeng Xu

doi:10.3389/fnins.2023.1212049

STNet: shape and texture joint learning through two-stream network for knowledge-guided image recognition

Front Neurosci. 2023 Jun 15:17:1212049. doi: 10.3389/fnins.2023.1212049. eCollection 2023.

Authors

Xijing Wang^#¹, Hongcheng Han^#¹, Mengrui Xu^{1

2}, Shengpeng Li¹, Dong Zhang^{1

3}, Shaoyi Du¹, Meifeng Xu⁴

Affiliations

¹ National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center for Visual Information and Applications, Institute of Artificial Intelligence and Robotics, Xi'an Jiaotong University, Xi'an, China.
² The School of Software Engineering, Xi'an Jiaotong University, Xi'an, China.
³ The School of Automation Science and Engineering, Xi'an Jiaotong University, Xi'an, China.
⁴ The Second Affiliated Hospital of Xi'an Jiaotong University (Xibei Hospital), Xi'an, China.

^# Contributed equally.

Abstract

Introduction: The human brain processes shape and texture information separately through different neurons in the visual system. In intelligent computer-aided imaging diagnosis, pre-trained feature extractors are commonly used in various medical image recognition methods, common pre-training datasets such as ImageNet tend to improve the texture representation of the model but make it ignore many shape features. Weak shape feature representation is disadvantageous for some tasks that focus on shape features in medical image analysis.

Methods: Inspired by the function of neurons in the human brain, in this paper, we proposed a shape-and-texture-biased two-stream network to enhance the shape feature representation in knowledge-guided medical image analysis. First, the two-stream network shape-biased stream and a texture-biased stream are constructed through classification and segmentation multi-task joint learning. Second, we propose pyramid-grouped convolution to enhance the texture feature representation and introduce deformable convolution to enhance the shape feature extraction. Third, we used a channel-attention-based feature selection module in shape and texture feature fusion to focus on the key features and eliminate information redundancy caused by feature fusion. Finally, aiming at the problem of model optimization difficulty caused by the imbalance in the number of benign and malignant samples in medical images, an asymmetric loss function was introduced to improve the robustness of the model.

Results and conclusion: We applied our method to the melanoma recognition task on ISIC-2019 and XJTU-MM datasets, which focus on both the texture and shape of the lesions. The experimental results on dermoscopic image recognition and pathological image recognition datasets show the proposed method outperforms the compared algorithms and prove the effectiveness of our method.

Keywords: brain-like information processing; computer-aided diagnosis; feature fusion; image recognition; joint learning; two-stream network.

Grants and funding

This work was supported by the National Key Research and Development Program of China under Grant No. 2017YFA0700800, the Natural Science Basic Research Plan in Shaanxi Province of China under Grant No. 2022JM-324, the Social Science Foundation of Shaanxi Province of China under Grant No. 2021K014, and the Key Project of Shaanxi Province under Grant No. 2018ZDCXLGY-06-07.