The continual spread of novel coronavirus disease 2019 (COVID-19) is caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), posing a severe threat to the health worldwide. The main protease (Mpro, alias 3CLpro) of SARS-CoV-2 is a crucial enzyme for the maturation of viral particles and is a very attractive target for designing drugs to treat COVID-19. Here, we propose a multiple conformation-based virtual screening strategy to discover inhibitors that can target SARS-CoV-2 Mpro. Based on this strategy, nine Mpro structures and a protein mimetics library with 8960 commercially available compounds were prepared to carry out ensemble docking for the first time. Five of the nine structures are apo forms presented in different conformations, whereas the other four structures are holo forms complexed with different ligands. The surface plasmon resonance assay revealed that 6 out of 49 compounds had the ability to bind to SARS-CoV-2 Mpro. The fluorescence resonance energy transfer experiment showed that the biochemical half-maximal inhibitory concentration (IC50) values of the six compounds could hamper Mpro activities ranged from 0.69 ± 0.05 to 2.05 ± 0.92 μM. Evaluation of antiviral activity using the cell-based assay indicated that two compounds (Z1244904919 and Z1759961356) could strongly inhibit the cytopathic effect and reduce replication of the living virus in Vero E6 cells with the half-maximal effective concentrations (EC50) of 4.98 ± 1.83 and 8.52 ± 0.92 μM, respectively. The mechanism of the action for the two inhibitors were further elucidated at the molecular level by molecular dynamics simulation and subsequent binding free energy analysis. As a result, the discovered noncovalent reversible inhibitors with novel scaffolds are promising antiviral drug candidates, which may be used to develop the treatment of COVID-19.