Advances in genetics and developmental biology are revealing the relationship between genotype and dental phenotype (G:P), providing new approaches for how paleontologists assess dental variation in the fossil record. Our aim was to understand how the method of trait definition influences the ability to reconstruct phylogenetic relationships and evolutionary history in the Cercopithecidae, the Linnaean Family of monkeys currently living in Africa and Asia. We compared the two-dimensional assessment of molar size (calculated as the mesiodistal length of the crown multiplied by the buccolingual breadth) to a trait that reflects developmental influences on molar development (the inhibitory cascade, IC) and two traits that reflect the genetic architecture of postcanine tooth size variation (defined through quantitative genetic analyses: MMC and PMM). All traits were significantly influenced by the additive effects of genes and had similarly high heritability estimates. The proportion of covariate effects was greater for two-dimensional size compared to the G:P-defined traits. IC and MMC both showed evidence of selection, suggesting that they result from the same genetic architecture. When compared to the fossil record, Ancestral State Reconstruction using extant taxa consistently underestimated MMC and PMM values, highlighting the necessity of fossil data for understanding evolutionary patterns in these traits. Given that G:P-defined dental traits may provide insight to biological mechanisms that reach far beyond the dentition, this new approach to fossil morphology has the potential to open an entirely new window onto extinct paleobiologies. Without the fossil record, we would not be able to grasp the full range of variation in those biological mechanisms that have existed throughout evolution.
Keywords: Cercopithecidae; dentition; evolution; genotype:phenotype mapping; monkeys; phylogeny; primates.