Many previous neuroimaging studies have shown that the early visual cortex of the early blind (EB) exhibits significant functional plasticity. However, only few previous studies have addressed the question whether or not such functional plasticity is accompanied by, and even related to, structural plasticity. In this study, we acquired high-resolution whole-brain anatomical magnetic resonance images form 14 Chinese EB adults, who lost sight before 6 years of age, and 16 age/gender-matched normal-sighted controls (SC), and compared pixel-by-pixel the gray matter (GM) and white matter (WM) volumes between the two groups with voxel-based morphometry. The results showed that, relative to the SC, the EB exhibits significantly reduced WM volumes in the optic tract and optic radiation and significant GM losses in the early visual cortex. The reduction of WM volume in the optic radiation of the EB was found be modulated by both the age at blindness onset and the duration of blindness. The reduction of GM volume in the early visual cortex of the EB appeared to be unaffected by the age at blindness onset. However, it was found in localized regions of the atrophic early visual cortex of the EB that the GM loss was progressive with aging and increasing duration of blindness. These results suggest that early visual deprivation induces significant structural plasticity in the optic pathway and early visual cortex of the EB, which likely occurs during both the critical period of early neurodevelopment and the course of persisted blindness later in life through activity-dependent mechanisms.