How to distinguish indolent from aggressive disease remains a great challenge in prostate cancer (PCa) management. Cullin 4B (CUL4B) is a scaffold protein and exhibits oncogenic activity in a variety of human malignancies. In this study, we utilized PCa tissue specimens, cell lines and xenograft models to determine whether CUL4B contributes to PCa progression and metastasis. Here, we show that CUL4B expression highly correlates with the aggressiveness of PCa. CUL4B expression promotes proliferation, epithelial-mesenchymal transition, and metastatic potential of PCa cells, whereas CUL4B knockdown inhibits. Mechanically, CUL4B positively regulates SOX4, a key regulator in PCa, through epigenetic silencing of miR-204. In turn, SOX4 upregulates CUL4B expression through transcriptional activation, thereby fulfilling a positive feedback loop. Clinically, CUL4B+/SOX4+ defines a subset of PCa patients with poor prognosis. Bioinformatics analysis further reveals that Wnt/ß-catenin activation signature is enriched in CUL4B+/SOX4+ patient subgroup. Intriguingly, Wnt inhibitors significantly attenuates oncogenic capacities of CUL4B in vitro and in vivo. Together, our study identifies CUL4B as a key modulator of aggressive PCa by a positive feedback loop that interacts with SOX4. This regulatory circuit may have a crucial role in PCa progression.