A woman's reproductive health and ability to have children directly affect numerous aspects of her life, from personal well-being and socioeconomic standing, to morbidity and lifespan. In turn, reproductive health depends on the development of correctly functioning ovaries, a process that starts early during fetal life. Early disruption to ovarian programming can have long-lasting consequences, potentially manifesting as disease much later in adulthood. A growing body of evidence suggests that exposure to chemicals early in life, including endocrine-disrupting chemicals, can cause a range of disorders later in life, such as those described in the ovarian dysgenesis syndrome hypothesis. In this Review, we discuss four specific time windows during which the ovary is particularly sensitive to disruption by exogenous insults: gonadal sex determination, meiotic division, follicle assembly and the first wave of follicle recruitment. To date, most evidence points towards the germ cell lineage being the most vulnerable to chemical exposure, particularly meiotic division and follicle assembly. Environmental chemicals and pharmaceuticals, such as bisphenols or mild analgesics (including paracetamol), can also affect the somatic cell lineages. This Review summarizes our current knowledge pertaining to environmental chemicals and pharmaceuticals, and their potential contributions to the development of ovarian dysgenesis syndrome. We also highlight knowledge gaps that need addressing to safeguard female reproductive health.