The present work reviews recent findings related to the action of steroidal (physiological) estrogens on normal mammary gland development and carcinogenesis, as well as effects of related environmental mediators (phyto- and xeno-estrogens), the role of which remains controversial. Orchestration by estrogen receptors (i.e. ERα and ERβ) and coregulators of growth, apoptosis and differentiation of epithelial cells, directed our analysis. The bidirectional coordination between epithelium and stroma in parallel with maintenance of stemness are also investigated. The relevance of nuclear and extranuclear localization of ERs and other eventual estrogen binding sites, mediating differential actions in regard to these various topics, is critically addressed to delineate the importance of direct and indirect activation procedures and delicate feedback loops (ligand-induced or/and cross-talk activation, respectively). The inclusion of the outlined regulatory concepts in drug design programs for the prevention and treatment of breast cancer may have potent effects.