Accurate identification and localization of the vertebrae in CT scans is a critical and standard pre-processing step for clinical spinal diagnosis and treatment. Existing methods are mainly based on the integration of multiple neural networks, and most of them use heatmaps to locate the vertebrae's centroid. However, the process of obtaining vertebrae's centroid coordinates using heatmaps is non-differentiable, so it is impossible to train the network to label the vertebrae directly. Therefore, for end-to-end differential training of vertebrae coordinates on CT scans, a robust and accurate automatic vertebral labeling algorithm is proposed in this study. First, a novel end-to-end integral regression localization and multi-label classification network is developed, which can capture multi-scale features and also utilize the residual module and skip connection to fuse the multi-level features. Second, to solve the problem that the process of finding coordinates is non-differentiable and the spatial structure of location being destroyed, an integral regression module is used in the localization network. It combines the advantages of heatmaps representation and direct regression coordinates to achieve end-to-end training and can be compatible with any key point detection methods of medical images based on heatmaps. Finally, multi-label classification of vertebrae is carried out to improve the identification rate, which uses bidirectional long short-term memory (Bi-LSTM) online to enhance the learning of long contextual information of vertebrae. The proposed method is evaluated on a challenging data set, and the results are significantly better than state-of-the-art methods (identification rate is 91.1% and the mean localization error is 2.2 mm). The method is evaluated on a new CT data set, and the results show that our method has good generalization.