Damage to radar absorbing materials (RAMs) reduces the stealth capabilities and battlefield survivability of the equipment. Research on RAM damage detection technology is key to outfield equipment maintenance. In this paper, an intelligent RAM damage detection method based on visual and microwave modalities is proposed. A compressed sensing planar-scanning microwave imaging method based on a range migration algorithm (RMA) imaging operator and fast Gaussian gridding nonuniform fast Fourier transform (FGG-NUFFT) is proposed, achieving high imaging quality and speed. A dual-modality, curved RAM dataset (DCR dataset) is constructed, composed of visual images and microwave images showing two kinds of damage: round shedding and strip cracks. A new dual-modality target detection model, the visual-microwave fusion network (VMFNet), is designed to detect RAM damage. Its mean average precision (mAP) reaches 81.87%, and its inference speed reaches 35.91 fps. A visual network (VisNet) and microwave network (MicNet) are designed as the backbone of VMFNet for extracting the visual and microwave features of RAMs. A path aggregation network (PANet) unit is designed to fuse the multiscale features of the two modalities, resulting in good retention of shallow-level features and high detection accuracy. The head contains different receptive fields and outputs three scales of detection results, effectively detecting damage of different sizes.