We systematically examined the evidence of declining sperm counts and the hypothesis that an increased exposure to environmental pollutants is responsible for such decline. Search engines, including PUBMED, MEDLINE, EMBASE, BIOSIS, and Cochrane library, were used to identify epidemiologic studies published from 1985 to 2013. We concluded that there is no enough evidence to confirm a worldwide decline in sperm counts. Also, there seems to be no scientific truth of a causative role for endocrine disruptors in the temporal decline of sperm production. Such assumptions are based on few meta-analyses and retrospective studies, while other well-conducted researches could not confirm these findings. We acknowledge that difficult-to-control confounding factors in the highly variable nature of semen, selection criteria, and comparability of populations from different time periods in secular-trend studies, the quality of laboratory methods for counting sperm, and apparently geographic variations in semen quality are the main issues that complicate the interpretation of the available evidence. Owing to the importance of this subject and the uncertainties still prevailing, there is a need not only for continuing monitoring of semen quality, reproductive hormones, and xenobiotics, but also for a better definition of fecundity.