Empathizing is defined as "the drive to identify another's mental states and to respond to these with an appropriate emotion" and systemizing is defined as "the drive to the drive to analyze and construct rule-based systems". While mean diffusivity (MD) has been robustly associated with several cognitive traits and disorders related with empathizing and systemizing, its direct correlation with empathizing and systemizing remains to be investigated. We undertook voxel-by-voxel investigations of regional MD to discover microstructural correlates of empathizing, systemizing, and the discrepancy between them (D score: systemizing - empathizing). Whole-brain analyses of covariance revealed that across both sexes, empathizing was positively correlated with MD of (a) an anatomical cluster that primarily spreads in the areas in and adjacent to the left dorsolateral prefrontal cortex, left anterior to the middle cingulate cortex, and left insula and (b) an anatomical cluster of the left postcentral gyrus and left rolandic operculum. The former overlaps with positive MD correlates of cooperativeness. The D score and systemizing did not show significant correlations. In conclusion, while increased MD has generally been associated with reduced neural tissues and possibly area function, higher empathizing and cooperativeness were commonly reflected by greater MD values in areas (a) that mainly overlap with areas that play a key role in emotional salience and empathy. In addition, higher empathizing was correlated with greater MD values in areas (b) that play a key role in the mirror neuron system.