Remote sensing (RS) scene classification is a challenging task to predict scene categories of RS images. RS images have two main issues: large intraclass variance caused by large resolution variance and confusing information from large geographic covering area. To ease the negative influence from the above two issues. We propose a multigranularity multilevel feature ensemble network (MGML-FENet) to efficiently tackle the RS scene classification task in this article. Specifically, we propose multigranularity multilevel feature fusion branch (MGML-FFB) to extract multigranularity features in different levels of network by channel-separate feature generator (CS-FG). To avoid the interference from confusing information, we propose a multigranularity multilevel feature ensemble module (MGML-FEM), which can provide diverse predictions by full-channel feature generator (FC-FG). Compared to previous methods, our proposed networks have the ability to use structure information and abundant fine-grained features. Furthermore, through the ensemble learning method, our proposed MGML-FENets can obtain more convincing final predictions. Extensive classification experiments on multiple RS datasets (AID, NWPU-RESISC45, UC-Merced, and VGoogle) demonstrate that our proposed networks achieve better performance than previous state-of-the-art (SOTA) networks. The visualization analysis also shows the good interpretability of MGML-FENet.