To date, the existing terahertz super-resolution reconstruction methods based on deep-learning networks have achieved noteworthy success. However, the terahertz image degradation process needs to fully consider the blur and noise of the high-frequency part of the image during the network training process, and cannot be replaced simply by interpolation, which has high complexity. The terahertz degradation model is systematically investigated, and effectively solves the above problems by introducing the remaining channel mechanism into the deep-learning network. On the one hand, an image degradation model suitable for the terahertz imaging process is adopted for the images in the training dataset, which improves the accuracy of network training. On the other hand, the residual channel attention mechanism is introduced to realize the adaptive adjustment of the dependence between network channels, which results in the network being more focused on the restoration of high-frequency information, thereby supporting the extraction of high-frequency edge details in the image. In addition, experimental results demonstrate that this method successfully improves the peak signal-to-noise ratios, and offers clearer edge details and a better overall reconstruction effect. We believe that this work may provide a new possibility to improve the resolution of terahertz images.