microRNAs (miRNAs; miR) are a class of small non-coding RNA molecules, which are involved in the pathogenesis of human diseases through the negative regulation of gene expression. Previous studies have demonstrated that miR-509-3p is a novel miRNA associated with cell proliferation and migration in 786-O renal cell carcinoma (RCC) cells. However, the mechanism of action of miR-509-3p in RCC remains to be elucidated. The present study aimed to examine the functional role and mechanism of miR-509-3p in the development of RCC. The results demonstrated that the expression levels of miR-509-3p were downregulated in the 786-O and ACHN RCC cell lines compared with the normal tissues of 10 patients with RCC, as determined by reverse transcription-quantitative polymerase chain reaction. The mRNA expression levels of mitogen-activated protein kinase kinase kinase 8 (MAP3K8) were upregulated in the RCC cell lines. Functional investigations demonstrated that the overexpression of miR-509-3p inhibited the migration and proliferation of the RCC cells, as determined by wound scratch and 3-(4,5-dimethylthiazol-2-yl)-2,5-diphenyltetrazolium bromide assays. Luciferase reporter assays revealed that the overexpression of miR-509-3p reduced the transcriptional activity of MAP3K8. Furthermore, the present study demonstrated that the ectopic transfection of miR-509-3p led to a significant reduction in the mRNA and protein expression levels of MAP3K8 in the RCC cells. Finally, knockdown of MAP3K8 inhibited the migration and proliferation of the RCC cells. Therefore, the results of the present study demonstrated that the miR-509-3p RCC suppressor was a significant regulator of the MAP3K8 oncogene, suggesting that it may have a potential therapeutic role in the treatment of RCC.