A cross-modal deep metric learning model for disease diagnosis based on chest x-ray images

Yufei Jin; Huijuan Lu; Zhao Li; Yanbin Wang

doi:10.1007/s11042-023-14790-7

A cross-modal deep metric learning model for disease diagnosis based on chest x-ray images

Multimed Tools Appl. 2023 Mar 15:1-22. doi: 10.1007/s11042-023-14790-7. Online ahead of print.

Authors

Yufei Jin^{1

2}, Huijuan Lu^{1

2}, Zhao Li³, Yanbin Wang⁴

Affiliations

¹ China JiLiang University, Hangzhou, 310018 Zhejiang China.
² Key Laboratory of Electromagnetic Wave Information Technology and Metrology of Zhejiang Province, Hangzhou, 310018 Zhejiang China.
³ Zhejiang University, Hangzhou, 310018 Zhejiang China.
⁴ College of Computer Science and Technology, Zhejiang University, Hangzhou, 310018 Zhejiang China.

Abstract

The emergence of unknown diseases is often with few or no samples available. Zero-shot learning and few-shot learning have promising applications in medical image analysis. In this paper, we propose a Cross-Modal Deep Metric Learning Generalized Zero-Shot Learning (CM-DML-GZSL) model. The proposed network consists of a visual feature extractor, a fixed semantic feature extractor, and a deep regression module. The network belongs to a two-stream network for multiple modalities. In a multi-label setting, each sample contains a small number of positive labels and a large number of negative labels on average. This positive-negative imbalance dominates the optimization procedure and may prevent the establishment of an effective correspondence between visual features and semantic vectors during training, resulting in a low degree of accuracy. A novel weighted focused Euclidean distance metric loss is introduced in this regard. This loss not only can dynamically increase the weight of hard samples and decrease the weight of simple samples, but it can also promote the connection between samples and semantic vectors corresponding to their positive labels, which helps mitigate bias in predicting unseen classes in the generalized zero-shot learning setting. The weighted focused Euclidean distance metric loss function can dynamically adjust sample weights, enabling zero-shot multi-label learning for chest X-ray diagnosis, as experimental results on large publicly available datasets demonstrate.

Keywords: Chest X-ray; Cross-modal; Deep metric learning; Generalized zero-shot learning; Multi-label classification.

© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2023, Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.