The effects of genetic polymorphisms on muscle structure and function remain elusive. The present study tested for possible associations of 16 polymorphisms (across ten candidate genes) with fittness and skeletal muscle phenotypes in 17- to 37-year-old healthy Caucasian male endurance (n = 86), power/strength (n = 75) and team athletes (n = 60), and non-athletes (n = 218). Skeletal muscle function was measured with eight performance tests covering multiple aspects of muscular fitness. Along with body mass and height, the upper arm and limb girths, and maximal oxygen uptake were measured. Genotyping was conducted on DNA extracted from blood. Of the 16 polymorphisms studied, nine (spanning seven candidate genes and four gene families/signalling pathways) were independently associated with at least one skeletal muscle fitness measure (size or function, or both) measure and explained up to 4.1% of its variation. Five of the studied polymorphisms (activin- and adreno-receptors, as well as myosine light chain kinase 1) in a group of one to three combined with body height, age and/or group explained up to 20.4% of the variation of muscle function. ACVR1B (rs2854464) contributed 2.0-3.6% to explain up to 14.6% of limb proximal girths. The G allele (genotypes AG and GG) of the ACVR1B (rs2854464) polymorphism was significantly overrepresented among team (60.4%) and power (62.0%) athletes compared to controls (52.3%) and endurance athletes (39.2%), and G allele was also most consistently/frequently associated with muscle size and power. Overall, the investigated polymorphisms determined up to 4.1% of the variability of muscular fitness in healthy young humans.