Emotion recognition is known to change with age, but associations between the change and brain atrophy are not well understood. In the current study atrophied brain regions associated with emotion recognition were investigated in elderly and younger participants. Group comparison showed no difference in emotion recognition score, while the score was associated with years of education, not age. We measured the gray matter volume of 18 regions of interest including the bilateral precuneus, supramarginal gyrus, orbital gyrus, straight gyrus, superior temporal sulcus, inferior frontal gyrus, insular cortex, amygdala, and hippocampus, which have been associated with social function and emotion recognition. Brain reductions were observed in elderly group except left inferior frontal gyrus, left straight gyrus, right orbital gyrus, right inferior frontal gyrus, and right supramarginal gyrus. Path analysis was performed using the following variables: age, years of education, emotion recognition score, and the 5 regions that were not different between the groups. The analysis revealed that years of education were associated with volumes of the right orbital gyrus, right inferior frontal gyrus, and right supramarginal gyrus. Furthermore, the right supramarginal gyrus volume was associated with the emotion recognition score. These results suggest that the amount of education received contributes to maintain the right supramarginal gyrus volume, and indirectly affects emotion recognition ability.