Motor imagery (MI) electroencephalogram (EEG) signals have an important role in brain-computer interface (BCI) research. However, effectively decoding these signals remains a problem to be solved. Traditional EEG signal decoding algorithms rely on parameter design to extract features, whereas deep learning algorithms represented by convolution neural network (CNN) can automatically extract features, which is more suitable for BCI applications. However, when EEG data is taken as input in raw time series, traditional 1D-CNNs are unable to acquire both frequency domain and channel association information. To solve this problem, this study proposes a novel algorithm by inserting two modules into CNN. One is the Filter Band Combination (FBC) Module, which preserves as many frequency domain features as possible while maintaining the time domain characteristics of EEG. Another module is Multi-View structure that can extract features from the output of FBC module. To prevent over fitting, we used a cosine annealing algorithm with restart strategy to update the learning rate. The proposed algorithm was validated on the BCI competition dataset and the experiment dataset, using accuracy, standard deviation, and kappa coefficient. Compared with traditional decoding algorithms, our proposed algorithm achieved an improvement of the maximum average correct rate of 6.6% on the motion imagery 4-classes recognition mission and 11.3% on the 2-classes classification task.