In recent years, object detection approaches using deep convolutional neural networks (CNNs) have derived major advances in normal images. However, such success is hardly achieved with rainy images due to lack of visibility. Aiming to bridge this gap, in this article, we present a novel selective features absorption network (SFA-Net) to improve the performance of object detection not only in rainy weather conditions but also in favorable weather conditions. SFA-Net accomplishes this objective by utilizing three subnetworks, where the feature selection subnetwork is concatenated with the object detection subnetwork through the feature absorption subnetwork to form a unified model. To promote further advancement in object detection impaired by rain, we propose a large-scale rainy image dataset, named srRain, which contains both synthetic rainy images and real-world rainy images for training and testing purposes. srRain is comprised of 25 900 rainy images depicting diverse driving scenarios in the presence of rain with a total of 181 164 instances interpreting five common object categories. Experimental results display that our SFA-Net reaches the highest mean average precision (mAP) of 77.53% on a normal image set, 62.52% on a synthetic rainy image set, 37.34% on a collected natural rainy image set, and 32.86% on a published real rainy image set, surpassing current state-of-the-art object detectors and the combination of image deraining and object detection models while retaining a high speed.