Increasing lines of evidence show that the malignant behavior of cancer is not exclusively attributable to cancer cells but also radically influenced by cancerous stroma activity and controlled through various mechanisms by the microenvironment. In addition to structural components, such as the extracellular matrix, stromal cells, such as macrophages, endothelial cells, and specifically cancer-associated fibroblasts (CAFs), have attracted substantial attention over recent decades. CAFs provide routes for aggressive carcinomas and contribute to invasion and metastasis through the biochemical alteration and regulation of cancer-related pathways. However, another facet of CAFs that has been neglected by numerous studies is that CAFs might serve as a negative regulator of cancer progression under certain circumstances. The various origins of CAFs, the diverse tissues in which they reside and their interactions with different cancer cells appear to be responsible for this inconsistency. This review summarizes the latest knowledge regarding CAF heterogeneity and offers a novel perspective and a beneficial approach for obtaining an improved understanding of CAFs.