Childhood socioeconomic status is robustly associated with various children's cognitive factors and neural mechanisms. Here we show the association of childhood socioeconomic status with psychometric intelligence and mean diffusivity and fractional anisotropy using diffusion tensor imaging at the baseline experiment (N = 285) and longitudinal changes in these metrics after 3.0 ± 0.3 years (N = 223) in a large sample of normal Japanese children (mean age = 11.2 ± 3.1 years). After correcting for confounding factors, cross-sectional and longitudinal analyses show that higher childhood socioeconomic status is associated with greater baseline and baseline to follow-up increase of psychometric intelligence and mean diffusivity in areas around the bilateral fusiform gyrus. These results demonstrate that higher socioeconomic status is associated with higher psychometric intelligence measures and altered microstructural properties in the fusiform gyrus which plays a key role in reading and letter recognition and further augmentation of such tendencies during development. Definitive conclusions regarding the causality of these relationships requires intervention and physiological studies. However, the current findings should be considered when developing and revising policies regarding education.