The aim of our study was to elucidate whether specific patterns of gray matter loss were associated with apolipoprotein E ε4 (APOE ε4) and microtubule-associated protein tau (MAPT)-H1) genetic variants in subjects with mild cognitive impairment (MCI) at a baseline visit. Gray matter voxel-based morphometry analysis of T1 magnetic resonance imaging scans were performed in 65 amnestic-MCI subjects. MCI APOE ε4 carriers compared with non-carriers showed increased brain atrophy in right hippocampus and rostral amygdala, superior and middle temporal gyrus, and right parietal operculum, including inferior frontal gyrus, inferior parietal, and supramarginal gyrus. MAPT-H1/H1 MCI carriers showed an increased bilateral atrophy in superior frontal gyri (including frontal eye fields and left prefrontal cortex) and precentral gyrus but also unilateral left atrophy in the inferior temporal gyrus and calcarine gyrus. In addition, MCI subjects carrying both APOE ε4 and MAPT-H1/H1 variants showed gray matter loss in the supplementary motor area and right pre- and postcentral gyri. The effect of APOE ε4 on gray matter loss in right hippocampus suggests that, at least in some AD sub-types, the neuronal vulnerability could be increased in the right hemisphere. The pattern of frontal gray matter loss observed among MCI MAPT H1/H1 carriers has also been found in other tauopathies, suggesting that MCI may share etiological factors with other tauopathies. Frontal and parietal cortex vulnerability was found when adding MAPT H1/H1 and APOE ε4 effects, suggesting a synergistic effect of these variants. These results could be due to changes in APOE ε4 and MAPT expression.