This study aimed to explore the association between body mass index (BMI), other anthropometric indexes and semen quality in a general male population in Taiwan. In this cross-sectional cohort study, the study cohort consisted of 7941 healthy male individuals aged 18 years or older who participated in a standard medical screening program run by a private firm from January 2008 to May 2013. Semen parameters including sperm concentration (SC), total sperm motility (TSM), progressive motility (PRM), and normal sperm morphology (NSM) were recorded. Anthropometric indexes including BMI, waist circumference (WC), hip circumference (HC), waist-to-hip ratio (WHR), waist-to-height ratio (WHtR) and body fat percentage were measured. A total of 7630 men were enrolled for the final analysis, of whom 68.5% had a normal weight distribution and 31.4% were overweight or obese. Total sperm motility, progressive motility, normal sperm morphology and sperm concentration showed a statistically linear decline with increasing age (p < 0.001, p < 0.001, p < 0.001 and p = 0.004). Sperm concentration showed a significantly negatively linear association with BMI (p = 0.005), and normal sperm morphology showed an inverse association with BMI and waist-to-height ratio (p < 0.001 and p = 0.004). The prevalence of abnormal total sperm motility, progressive motility, normal sperm morphology and sperm concentration increased with increasing age (p = 0.011, p < 0.001, p < 0.001 and p = 0.002). Lower normal sperm morphology and sperm concentration were associated with increasing body adiposity (p<0.05). No relationship between obesity and sperm motility was identified.