The biology of caveolin-1 (Cav1)/caveolae is intimately linked to actin dynamics and adhesion receptors. Caveolar domains are organized in hierarchical levels of complexity from curved or flattened caveolae to large, higher-order caveolar rosettes. We report that stress fibers controlled by Abl kinases and mDia1 determine the level of caveolar domain organization, which conditions the subsequent inward trafficking of caveolar domains induced upon loss of cell adhesion from the extracellular matrix. Abl-deficient cells have fewer stress fibers, a smaller pool of stress-fiber co-aligned Cav1 and increased clustering of Cav1/caveolae at the cell surface. Defective caveolar linkage to stress fibers prevents the formation of big caveolar rosettes upon loss of cell adhesion, correlating with a lack of inward trafficking. Live imaging of stress fibers and Cav1 showed that the actin-linked Cav1 pool loses its spatial organization in the absence of actin polymerization and is dragged and clustered by depolymerizing filaments. We identified mDia1 as the actin polymerization regulator downstream of Abl kinases that controls the stress-fiber-linked Cav1 pool. mDia1 knockdown results in Cav1/caveolae clustering and defective inward trafficking upon loss of cell adhesion. By contrast, cell elongation imposed by the excess of stress fibers induced by active mDia1 flattens caveolae. Furthermore, active mDia1 rescues the actin co-aligned Cav1 pool and Cav1 inward trafficking upon loss of adhesion in Abl-deficient cells. Thus, caveolar domain organization and trafficking are tightly coupled to adhesive and stress fiber regulatory pathways.