Fragile X syndrome (FXS) is a neurodevelopmental disorder that is caused by mutations in the FMR1 gene. Neuroanatomical alterations have been reported in both male and female individuals with FXS, yet the morphological underpinnings of these alterations have not been elucidated. In the current study, we found structural changes in both male and female rats that model FXS, some of which are similarly impaired in both sexes, including the superior colliculus and periaqueductal gray, and others that show sex-specific changes. The splenium of the corpus callosum, for example, was only impaired in males. We also found reduced axonal caliber in the splenium, offering a mechanism for its structural changes. Furthermore, we found that overall, male rats have higher brain-wide diffusion than female rats. Our results provide insight into which brain regions are vulnerable to a loss of Fmr1 expression and reveal an impairment at the level of the axon that could cause structural changes in white matter regions.