During the course of evolution, the human skeletal system has evolved rapidly leading to an incredible array of phenotypic diversity, including variations in height and bone mineral density. However, the genetic basis of this phenotypic diversity and the relatively rapid tempo of evolution have remained largely undocumented. Here, we discover that skeletal genes exhibit a significantly greater level of population differentiation among humans compared with other genes in the genome. The pattern is exceptionally evident at amino acid-altering sites within these genes. Divergence is greater between Africans and both Europeans and East Asians. In contrast, relatively weak differentiation is observed between Europeans and East Asians. SNPs with higher levels of differentiation have correspondingly higher derived allele frequencies in Europeans and East Asians. Thus, it appears that positive selection has operated on skeletal genes in the non-African populations and this may have been initiated with the human colonization of Eurasia. In conclusion, we provide genetic evidence supporting the rapid evolution of the human skeletal system and the associated diversity of phenotypes.