Observations of the disproportionate incidence of depression in women compared with men have long preceded the recent explosion of interest in sex differences. Nonetheless, the source and implications of this epidemiologic sex difference remain unclear, as does the practical significance of the multitude of sex differences that have been reported in brain structure and function. In this article, we attempt to provide a framework for thinking about how sex and reproductive hormones (particularly estradiol as an example) might contribute to affective illness. After briefly reviewing some observed sex differences in depression, we discuss how sex might alter brain function through hormonal effects (both organizational (programmed) and activational (acute)), sex chromosome effects, and the interaction of sex with the environment. We next review sex differences in the brain at the structural, cellular, and network levels. We then focus on how sex and reproductive hormones regulate systems implicated in the pathophysiology of depression, including neuroplasticity, genetic and neural networks, the stress axis, and immune function. Finally, we suggest several models that might explain a sex-dependent differential regulation of affect and susceptibility to affective illness. As a disclaimer, the studies cited in this review are not intended to be comprehensive but rather serve as examples of the multitude of levels at which sex and reproductive hormones regulate brain structure and function. As such and despite our current ignorance regarding both the ontogeny of affective illness and the impact of sex on that ontogeny, sex differences may provide a lens through which we may better view the mechanisms underlying affective regulation and dysfunction.