Precision medical image hash retrieval by interpretability and feature fusion

Abstract

Background and objective: To address the problem of low accuracy of medical image retrieval due to high inter-class similarity and easy omission of lesions, a precision medical image hash retrieval method combining interpretability and feature fusion is proposed, taking chest X-ray images as an example.

Methods: Firstly, the DenseNet-121 network is pre-trained on a large dataset of medical images without manual annotation using the comparison to learn (C2L) method to obtain a backbone network model containing more medical representations with training weights. Then, a global network is constructed by using global image learning to acquire an interpretable saliency map as attention mechanisms, which can generate a mask crop to get a local discriminant region. Thirdly, the local discriminant regions are used as local network inputs to obtain local features, and the global features are used with the local features by dimension in the pooling layer. Finally, a hash layer is added between the fully connected layer and the classification layer of the backbone network, defining classification loss, quantization loss and bit-balanced loss functions to generate high-quality hash codes. The final retrieval result is output by calculating the similarity metric of the hash codes.

Results: Experiments on the Chest X-ray8 dataset demonstrate that our proposed interpretable saliency map can effectively locate focal regions, the fusion of features can avoid information omission, and the combination of three loss functions can generate more accurate hash codes. Compared with the current advanced medical image retrieval methods, this method can effectively improve the accuracy of medical image retrieval.

Conclusions: The proposed hash retrieval approach combining interpretability and feature fusion can effectively improve the accuracy of medical image retrieval which can be potentially applied in computer-aided-diagnosis systems.

Keywords: Attention mechanisms; Deep hashing; Feature fusion; Interpretability; Medical image retrieval.