Obesity remains a major global health concern. Understanding the metabolic influences of the obesity epidemic in the human population on maintenance of a healthy weight and metabolic profile is still of great significance. The importance and role of white adipose tissue has been long established, particularly with excess adiposity. Brown adipose tissue (BAT), however, has only recently been shown to contribute significantly to the metabolic signature of mammals outside the previously recognised role in small mammals and neonates. BAT's detection in adults has led to a renewed interest and is now considered to be a potential therapeutic target to prevent excess white fat accumulation in obesity, a theory further promoted by the recent discovery of beige fat. Adipose tissue distribution varies significantly between genders. Pre-menopausal females often show enhanced lower and peripheral fat deposition in adiposity deposition compared to the male profile of central and visceral fat accumulation with obesity. This sex disparity is partly attributed to the different effects of sex hormone profiles and interactions on the adipose tissue system. In this review, we explore this intricate relationship and show how modifications in the effects of sex hormones impact on both brown and white adipose tissues. We also discuss the impact of sex hormones on activation of the hypothalamic-pituitary-adrenal (HPA) axis and how the three pathways between adiposity, HPA and sex steroids can have a major contribution to the prevention or maintenance of obesity and therefore on overall health.