Owing to the development of computerized vision technology, object detection based on convolutional neural networks is being widely used in the field of bridge crack detection. However, these networks have limited utility in bridge crack detection because of low precision and poor real-time performance. In this study, an improved single-shot multi-box detector (SSD) called ISSD is proposed, which seamlessly combines the depth separable deformation convolution module (DSDCM), inception module (IM), and feature recalibration module (FRM) in a tightly coupled manner to tackle the challenges of bridge crack detection. Specifically, DSDCM was utilized for extracting the characteristic information of irregularly shaped bridge cracks. IM was designed to expand the width of the network, reduce network calculations, and improve network computing speed. The FRM was employed to determine the importance of each feature channel through learning, enhance the useful features according to their importance, and suppress the features that are insignificant for bridge crack detection. The experimental results demonstrated that ISSD is effective in bridge crack detection tasks and offers competitive performance compared to state-of-the-art networks.