Quantitative magnetic resonance imaging (MRI) investigations of brain anatomy in children and young adults with Down syndrome (DS) are limited, with no diffusion tensor imaging (DTI) studies covering that age range. We used DTI-driven tensor based morphometry (DTBM), a novel technique that extracts morphometric information from diffusion data, to investigate brain anatomy in 15 participants with DS and 15 age- and sex-matched typically developing (TD) controls, ages 6-24 years (mean age ~17 years). DTBM revealed marked hypoplasia of cerebellar afferent systems in DS, including fronto-pontine (middle cerebellar peduncle) and olivo-cerebellar (inferior cerebellar peduncle) connections. Prominent gray matter hypoplasia was observed in medial frontal regions, the inferior olives, and the cerebellum. Very few abnormalities were detected by classical diffusion MRI metrics, such as fractional anisotropy and mean diffusivity. Our results highlight the potential importance of cerebro-cerebellar networks in the clinical manifestations of DS and suggest a role for DTBM in the investigation of other brain disorders involving white matter hypoplasia or atrophy.