Estrogen receptor α (ERα) is a successful target for ER-positive breast cancer and also reported to be relevant in many other diseases. Selective estrogen receptor modulators (SERMs) make a good therapeutic effect in clinic. Because of the drug resistance and side effects of current SERMs, the discovery of new SERMs is given more and more attention. Virtual screening is a validated method to high effectively to identify novel bioactive small molecules. Ligand-based machine learning methods and structure-based molecular docking were first performed for identification of ERα antagonist from in-house natural product library. Naive Bayesian and recursive partitioning models with two kinds of descriptors were built and validated based on training set, test set, and external test set and then were utilized for distinction of active and inactive compounds. Totally, 162 compounds were predicted as ER antagonists and were further evaluated by molecular docking. According to docking score, we selected 8 representative compounds for both ERα competitor assay and luciferase reporter gene assay. Genistein, daidzein, phloretin, ellagic acid, ursolic acid, (-)-epigallocatechin-3-gallate, kaempferol, and naringenin exhibited different levels for antagonistic activity against ERα. These studies validated the feasibility of machine learning methods for predicting bioactivities of ligands and provided better insight into the natural products acting as estrogen receptor modulator, which are important lead compounds for future new drug design.