Sex-limited selection can moderate the elimination of deleterious mutations from the population and contribute to the high prevalence of common human diseases. Accordingly, deleterious mutations in autosomal genes that are exclusively expressed in only one of the sexes undergo sex-limited selection and can reach higher frequencies than mutations similarly selected in both sexes. Here we show that the number of deleterious SNPs in genes exclusively expressed in men is twofold higher than in genes that are selected in both sexes. Additional analyses suggest that the increased number of damaging mutations we found in male-specific genes is due to reduced selection in females. These results are noteworthy since many of these male-specific genes are known to be crucial for male reproduction, and are thus likely to be under strong purifying selection. We suggest that inheritance of male-infertility-causative mutations through unaffected female lineages contributes to the high incidence of male infertility.