The stroma-vascular fraction (SVF) of adipose tissue (AT) is a heterogeneous cell fraction composed of progenitor cells, endothelial cells, and immune cells. SVF plays a key role in AT homeostasis and growth as well as in obesity-associated pathologies. The SVF cell composition and phenotype are distinct according to AT location and adiposity. Such discrepancies influence AT function and are involved in obesity-associated disorders such as chronic inflammation. Investigations performed in recent years in rodents and humans provided evidence that the stroma-vascular cells contribute to the conversion of steroid hormones in AT and are also steroid targets. This review describes the link between steroids and SVF depending on gender, adiposity, and AT location and highlights the potential role of sex and corticosteroid hormones in adipogenesis, angiogenesis, and their contributions in AT inflammation.