In this paper, we aim at finding the smallest set of EEG channels that can ensure highly accurate classification of motor imagery (MI) dataset and maintain the optimum Kappa score. Non-negative matrix factorization (NMF) is used for important and discriminant EEG channel selection. Further, the theory of Riemannian geometry in the manifold of covariance matrices is used for feature extraction. At last, the neighborhood component feature selection (NCFS) algorithm is used to select the small subset of important features from the given set of features. The significance of the proposed work is two-fold: 1) it greatly reduces the time complexity and the amount of overfitting by reducing the unnecessary EEG channels and redundant features. 2) it increases the classification accuracy of the model by selecting only subject-specific EEG channels. The proposed algorithm is tested on BCI Competition IV,2a dataset to validate the performance. The proposed approach has achieved 77.91% average classification accuracy and 0.626 mean Kappa score.