Magnetic resonance imaging has become an important noninvasive technique to gain insight into fetal brain development. Its capabilities go beyond ultrasound when diagnosing high-risk pregnancies. To summarize observations across a population in magnetic resonance imaging studies, reference systems such as atlases that establish correspondences across a cohort are key. In this article, we review the evolution of atlas-building methods in light of their relevance, limitations, and benefits for the modeling of human brain development. Starting with single anatomical templates to which brain scans where mapped to such as Talairach and Montreal Neurological Institute space, we explore the uses of atlases as a means to establish correspondences across a cohort and as a model that captures the population characteristics of the cases the atlas is built from. We discuss methods that capture features of increasingly heterogeneous populations and approaches that are able to generalize with only minimal annotation. The main focus of this review are methods that explicitly model the variability in the population with regard to time, such as in the modeling of disease progression and brain development. We highlight the applicability and limitations of state-of-the art approaches, how insights from the study of disease progression are helpful in developmental studies, and point to the directions of future research that is still necessary.