A general trait of living cells is their ability to exert contractile stresses on their surroundings and thus respond to substrate rigidity. At the cellular scale, this response affects cell shape, polarity, and ultimately migration. The regulation of cell shape together with rigidity sensing remains largely unknown. In this article we show that both substrate rigidity and cell shape contribute to drive actin organization and cell polarity. Increasing substrate rigidity affects bulk properties of the actin cytoskeleton by favoring long-lived actin stress fibers with increased nematic interactions, whereas cell shape imposes a local alignment of actin fibers at the cell periphery.