With the large-scale application of composite materials in military aircraft, various composite material detection technologies with infrared nondestructive and ultrasonic nondestructive testing as the core have played an important role in detecting composite material component damage in military aircraft. At present, the damage of composite materials is mostly recognized manually, which is time-consuming, laborious, and inefficient. It can effectively improve detection efficiency and accuracy by using intelligent detection methods to detect and recognize damage. Moreover, the effect of infrared detection is significantly reduced with increasing detection depth, while ultrasonic detection has shallow-blind areas. A cascade fusion R-CNN network is proposed in order to comprehensively identify composite material damage. This network realizes the intelligent fusion recognition of infrared and ultrasonic damage images of composite materials. The network is based on a cascade R-CNN network, using fusion modules and BiFPN for improvement. For the infrared image and ultrasonic C-scan image data set established in this paper, the algorithm can identify the type and location of damage detected by infrared and ultrasonic testing. Its recognition accuracy is 99.3% and mean average precision (mAP) is 90.4%. In the detection process, the characteristics of infrared and ultrasonic images are used to realize the recognition of damage depth. Compared to SSD, YOLOv4, faster R-CNN and cascade R-CNN, the network proposed in this paper is better and more effective.