Objective: Mapping CT number to material property dominates the proton range uncertainty. This work aims to develop a physics-constrained deep learning-based multimodal imaging (PDMI) framework to integrate physics, deep learning, MRI, and advanced dual-energy CT (DECT) to derive accurate patient mass density maps.
Methods: Seven tissue substitute MRI phantoms were used for validation including adipose, brain, muscle, liver, skin, spongiosa, 45% hydroxyapatite (HA) bone. MRI images were acquired using T1 weighted Dixon and T2 weighted short tau inversion recovery sequences. Training inputs are from MRI and twin-beam dual-energy images acquired at 120 kVp with gold/tin filters. The feasibility investigation included an empirical model and four residual networks (ResNet) derived from different training inputs and strategies by PDMI framework. PRN-MR-DE and RN-MR-DE denote ResNet (RN) trained with and without a physics constraint (P) using MRI (MR) and DECT (DE) images. PRN-DE stands for RN trained with a physics constraint using only DE images. A retrospective study using institutional patient data was also conducted to investigate the feasibility of the proposed framework.
Results: For the tissue surrogate study, PRN-MR-DE, PRN-DE, and RN-MR-DE result in mean mass density errors: -0.72%/2.62%/-3.58% for adipose; -0.03%/-0.61%/-0.18% for muscle; -0.58%/-1.36%/-4.86% for 45% HA bone. The retrospective patient study indicated that PRN-MR-DE predicted the densities of soft tissue and bone within expected intervals based on the literature survey, while PRN-DE generated large density deviations.
Conclusion: The proposed PDMI framework can generate accurate mass density maps using MRI and DECT images. The supervised learning can further enhance model efficacy, making PRN-MR-DE outperform RN-MR-DE. The patient investigation also shows that the framework can potentially improve proton range uncertainty with accurate patient mass density maps.
Advances in knowledge: PDMI framework is proposed for the first time to inform deep learning models by physics insights and leverage the information from MRI to derive accurate mass density maps.