The phenotypic association between fragile X syndrome (FXS) and autism is well established, but no studies have directly compared whole-brain anatomy between the two disorders. We performed voxel-based morphometry analyses of magnetic resonance imaging (MRI) scans on 10 individuals with FXS, 10 individuals with autism, and 10 healthy comparison subjects to identify volumetric changes in each disorder. Regional gray matter volumes within frontal, parietal, temporal, and cingulate gyri, as well as in the caudate nuclei and cerebellum, were larger in the FXS group relative to the autism group. In addition, volume increases in FXS were observed in frontal gyri and caudate nuclei compared to controls. The autism group exhibited volume increases in frontal and temporal gyri relative to the FXS group, and no volume increases relative to controls. Volumetric deficits relative to controls were observed in regions of the cerebellum for both groups, with additional deficits in parietal and temporal gyri for the FXS group. Our caudate nuclei and frontal gyri results may implicate dysfunction of frontostriatal circuitry in FXS. Cerebellar deficits suggest atypical development of the cerebellum contributing to the phenotype of both disorders, but further imply that unique cerebellar regions contribute to the phenotype of each disorder.