The object detection of remote sensing image often has low accuracy and high missed or false detection rate due to the large number of small objects, instance level noise and cloud occlusion. In this paper, a new object detection model based on SRGAN and YOLOV3 is proposed, which is called SR-YOLO. It solves the problems of SRGAN network sensitivity to hyper-parameters and modal collapse. Meanwhile, The FPN network in YOLOv3 is replaced by PANet, shortened the distance between the lowest and the highest layers, and the SR-YOLO model has strong robustness and high detection ability by using the enhanced path to enrich the characteristics of each layer. The experimental results on the UCAS-High Resolution Aerial Object Detection Dataset showed SR-YOLO has achieved excellent performance. Compared with YOLOv3, the average precision (AP) of SR-YOLO increased from 92.35% to 96.13%, the log-average miss rate (MR-2) decreased from 22% to 14%, and the Recall rate increased from 91.36% to 95.12%.