Previous studies by our group showed that Qianliening capsules (QC), a clinically proven effective traditional Chinese formulation that has long been used in the treatment of benign prostatic hyperplasia (BPH), is capable of inhibiting BPH in vivo and in vitro via the promotion of apoptosis, suppression of the EGFR/STAT3 signaling pathway and regulating the expression of sex hormones as well as their receptors. However, the mechanism of its anti-BPH activity has remained to be fully elucidated. The present study aimed to investigate the mechanism underlying the anti-proliferative effect of QC in vivo and in vitro. Castrated male Sprage-Dawley (SD) rats where subcutaneously injected with testosterone propionate and the WPMY-1 cell line was stimulated with basic fibroblast growth factor in order to generate BPH in vivo and in vitro separately, both of which were then subjected to QC treatment. Finasteride was used as a positive control drug for the in vivo study. In the present study, it was found that treatment with QC or finasteride significantly reduced the prostatic index (PI=prostate wet weight/body weight x 100) in a rat model of BPH (P<0.05). In addition, reverse transcription quantitative polymerase chain reaction (RT-PCR) and western blot analyses showed that QC or finasteride treatment significantly inhibited model construction-induced upregulation of expression of proliferating cell nuclear antigen, cyclin D1 and cyclin-dependent kinase 4 in prostatic tissues of rats with BPH (P<0.05). The in vitro study further proved that QC exhibited anti-proliferative properties via G1/S cell cycle arrest in the WPMY-1 cell line, as evidenced by colony formation, flow cytometric cell cycle, immunoblot and RT-PCR analyses. In conclusion, the present study demonstrated that inhibition of cell proliferation via G1/S cell cycle arrest may be one of the underlying mechanisms of the effect of QC on BPH.