Lead, cadmium, mercury, and arsenic, often referred to as "heavy metals", are toxic for wildlife, experimental animals, and humans. While experimental animal and human occupational studies with high exposure levels generally support an adverse role for these metals in human reproductive outcomes, information on the effects of low, environmentally-realistic exposure levels of these metals on male reproductive outcomes is limited. We review the literature on effects of exposure to low levels of these metals on measures of male fertility (semen quality and reproductive hormone levels) and provide supporting evidence from experimental and occupational studies. Potentially modifying effects of genetic polymorphisms on these associations are discussed. A brief review of the literature on the effects of three trace metals, copper, manganese, and molybdenum, that are required for human health, yet may also cause adverse reproductive effects, follows. Overall, there were few studies examining the effects of exposure to low levels of these metals on male reproductive health. For all metals, there were several well-designed studies with sufficient populations appropriately adjusted for potential confounders and many of these reported harmful effects. However, many studies lacked sufficient numbers of participants to be able to detect differences in outcomes between exposed and non-exposed individuals, did not clearly identify the source and characteristics of the participants, and did not control for other exposures that could alter or contribute to the outcomes. The evidence for the effects of low exposure was strongest for cadmium, lead, and mercury and less certain for arsenic. The potential modifying effects of genetic polymorphisms has not been fully explored. Additional studies on the reproductive effects of these toxic ubiquitous metals on male reproduction are required to expand the knowledge base and to resolve inconsistencies.