Bundle branch block (BBB) is one of the most common cardiac disorder, and can be detected by electro-cardiogram (ECG) signal in clinical practice. Conventional methods adopted some kinds of hand-craft features, whose discriminative power is relatively low. On the other hand, these methods were based on the supervised learning, which required the high cost heartbeat annotation in the training. In this paper, a novel end-to-end deep network was proposed to classify three types of heartbeat: right BBB (RBBB), left BBB (LBBB) and others with a multiple instance learning based training strategy. We trained the proposed method on the China Physiological Signal Challenge 2018 database (CPSC) and tested on the MIT-BIH Arrhythmia database (AR). The proposed method achieved an accuracy of 78.58%, and sensitivity of 84.78% (LBBB), 51.23% (others) and 99.72% (RBBB), better than the baseline methods. Experimental results show that our method would be a good choice for the BBB classification on the ECG dataset with record-level labels instead of heartbeat annotations.