Terahertz imaging technology has been gradually used in space communication, radar detection, aerospace and biomedical fields. Nevertheless, there are still some limits in terahertz image, such as single tone, fuzzy texture features, poor image resolution and less data, which seriously affect the application and popularization of Terahertz image technology in many fields. Traditional convolutional neural network (CNN) is an effective method for image recognition, but it is limited in highly blurred terahertz image recognition due to the great difference between terahertz image and traditional optical image. This paper presents a proven method for higher recognition rate of blurred terahertz images by using an improved Cross-Layer CNN model with different definition terahertz image dataset. Compared to employing clear image dataset, the accuracy of blurred image recognition can be improved from about 32% to 90% with different definition dataset. Meanwhile, the recognition accuracy of high blurred image can be improved by approximately 5% in contrast to the traditional CNN, which makes the higher recognition ability of neural network. It can be demonstrated that various types of blurred terahertz imaging data can be effectively identified by constructing different definition dataset combined with Cross-Layer CNN. A new method is proved to improve the recognition accuracy of terahertz imaging and application robustness in real scenarios.