Prostate cancer is a common malignancy of the male reproductive-urinary system. MDM2 is an oncogene, whose expression can be regulated by microRNA (miRNA). The present study investigated the expression and correlation of miRNA-509-5p and MDM2 to determine the mechanism of their function in invasion and migration of prostate cancer cells. RT-PCR was performed to detect the expression of miRNA-509-5p and MDM2 in tumor, tumor-adjacent, and normal tissues, obtained from prostate cancer patients, using the HGC-27 cell line as an in vitro model. Cultured HGC-27 cells were transfected with miRNA-509-5p mimics, miRNA-509-5p inhibitor, and mimic control. Expression levels of miRNA-509-5p and MDM2 were quantified by RT-PCR. Cell proliferation and invasion/migration were examined by the MTT and transwell assays, respectively. MiRNA-509-5p was significantly down-regulated in prostate cancer cells exhibiting high MDM2 mRNA levels. MiRNA mimic transfection elevated miRNA levels and suppressed MDM2 expression. With prolonged incubation time, the proliferation ratio and OD values of miRNA-509-5p mimic transfected cells decreased, along with decrease in cell migration and invasion. These results suggested that miRNA-509-5p negatively regulates MDM2 expression via targeting the 3'-UTR of genes. As a novel tumor suppressor, miRNA-509-5p in prostate cancer HGC-27 cells can suppress MDM2 expression and inhibit cell proliferation, invasion, and migration. Therefore, miRNA-509-5p could be used as a novel therapeutic agent in the treatment of prostate cancer.