Automatic Generation of Structured Radiology Reports for Volumetric Computed Tomography Images Using Question-Specific Deep Feature Extraction and Learning

Samira Loveymi; Mir Hossein Dezfoulian; Muharram Mansoorizadeh

doi:10.4103/jmss.JMSS_21_20

Automatic Generation of Structured Radiology Reports for Volumetric Computed Tomography Images Using Question-Specific Deep Feature Extraction and Learning

J Med Signals Sens. 2021 Jul 21;11(3):194-207. doi: 10.4103/jmss.JMSS_21_20. eCollection 2021 Jul-Sep.

Authors

Samira Loveymi¹, Mir Hossein Dezfoulian¹, Muharram Mansoorizadeh¹

Affiliation

¹ Department of Computer Engineering, Bu-Ali Sina University, Hamedan, Iran.

Abstract

Background: In today's modern medicine, the use of radiological imaging devices has spread at medical centers. Therefore, the need for accurate, reliable, and portable medical image analysis and understanding systems has been increasing constantly. Accompanying images with the required clinical information, in the form of structured reports, is very important, because images play a pivotal role in detect, planning, and diagnosis of different diseases. Report-writing can be exposure to error, tedious and labor-intensive for physicians and radiologists; to address these issues, there is a need for systems that generate medical image reports automatically and efficiently. Thus, automatic report generation systems are among the most desired applications.

Methods: This research proposes an automatic structured-radiology report generation system that is based on deep learning methods. Extracting useful and descriptive image features to model the conceptual contents of the images is one of the main challenges in this regard. Considering the ability of deep neural networks (DNNs) in soliciting informative and effective features as well as lower resource requirements, tailored convolutional neural networks and MobileNets are employed as the main building blocks of the proposed system. To cope with challenges such as multi-slice medical images and diversity of questions asked in a radiology report, our system develops volume-level and question-specific deep features using DNNs.

Results: We demonstrate the effectiveness of the proposed system on ImageCLEF2015 Liver computed tomography (CT) annotation task, for filling in a structured radiology report about liver CT. The results confirm the efficiency of the proposed approach, as compared to classic annotation methods.

Conclusion: We have proposed a question-specific DNNbased system for filling in structured radiology reports about medical images.

Keywords: Convolutional neural network; MobileNet; medical image analysis; radiology report generation.