Susceptibility induced distortion is a major artifact that affects the diffusion MRI (dMRI) data analysis. In the Human Connectome Project (HCP), the state-of-the-art method adopted to correct this kind of distortion is to exploit the displacement field from the B0 image in the reversed phase encoding images. However, both the traditional and learning-based approaches have limitations in achieving high correction accuracy in certain brain regions, such as brainstem. By utilizing the fiber orientation distribution (FOD) computed from the dMRI, we propose a novel deep learning framework named DistoRtion Correction Net (DrC-Net), which consists of the U-Net to capture the latent information from the 4D FOD images and the spatial transformer network to propagate the displacement field and back propagate the losses between the deformed FOD images. The experiments are performed on two datasets acquired with different phase encoding (PE) directions including the HCP and the Human Connectome Low Vision (HCLV) dataset. Compared to two traditional methods topup and FODReg and two deep learning methods S-Net and flow-net, the proposed method achieves significant improvements in terms of the mean squared difference (MSD) of fractional anisotropy (FA) images and minimum angular difference between two PEs in white matter and also brainstem regions. In the meantime, the proposed DrC-Net takes only several seconds to predict a displacement field, which is much faster than the FODReg method.