Prostate cancer represents the major cause of cancer-related death in men and patients frequently develop drug-resistance and metastatic disease. Most studies focus on hormone-resistance mechanisms related to androgen receptor mutations or to the acquired property of prostate cancer cells to over-activate signaling pathways. Tumor microenvironment plays a critical role in prostate cancer progression. However, the mechanism involving androgen/androgen receptor signaling in cancer associated fibroblasts and consequences for prostate cancer progression still remains elusive. We now report that prostate cancer associated fibroblasts express a transcriptional-incompetent androgen receptor. Upon androgen challenging, the receptor co-localizes with the scaffold protein filamin A in the extra-nuclear compartment of fibroblasts, thus mediating their migration and invasiveness. Cancer-associated fibroblasts move towards epithelial prostate cancer cells in 2D and 3D cultures, thereby inducing an increase of the prostate cancer organoid size. Androgen enhances both these effects through androgen receptor/filamin A complex assembly in cancer-associated fibroblasts. An androgen receptor-derived stapled peptide, which disrupts the androgen receptor/filamin A complex assembly, abolishes the androgen-dependent migration and invasiveness of cancer associated fibroblasts. Notably, the peptide impairs the androgen-induced invasiveness of CAFs in 2D models and reduces the overall tumor area in androgen-treated 3D co-culture. The androgen receptor in association with β1 integrin and membrane type-matrix metalloproteinase 1 activates a protease cascade triggering extracellular matrix remodeling. The peptide also impairs the androgen activation of this cascade. This study offers a potential new marker, the androgen receptor/filamin A complex, and a new therapeutic approach targeting intracellular pathways activated by the androgen/androgen receptor axis in prostate cancer-associated fibroblasts. Such a strategy, alone or in combination with conventional therapies, may allow a more efficient treatment of prostate cancer.