Although fibroblast heterogeneity is recognized in primary tumors, both its characterization in and its impact on metastases remain unknown. Here, combining flow cytometry, immunohistochemistry and RNA-sequencing on breast cancer samples, we identify four Cancer-Associated Fibroblast (CAF) subpopulations in metastatic lymph nodes (LN). Two myofibroblastic subsets, CAF-S1 and CAF-S4, accumulate in LN and correlate with cancer cell invasion. By developing functional assays on primary cultures, we demonstrate that these subsets promote metastasis through distinct functions. While CAF-S1 stimulate cancer cell migration and initiate an epithelial-to-mesenchymal transition through CXCL12 and TGFβ pathways, highly contractile CAF-S4 induce cancer cell invasion in 3-dimensions via NOTCH signaling. Patients with high levels of CAFs, particularly CAF-S4, in LN at diagnosis are prone to develop late distant metastases. Our findings suggest that CAF subset accumulation in LN is a prognostic marker, suggesting that CAF subsets could be examined in axillary LN at diagnosis.