Apolipoprotein epsilon4 (apoE4) has been strongly linked with Alzheimer's disease (AD) and contributes to several other neurological disorders. We investigated the influence of epsilon4 allele carrier status on the pattern of gray matter atrophy and disease severity in 51 patients with probable AD and 31 patients with behavioral variant frontotemporal dementia (bvFTD), compared with 56 healthy controls. Voxel-based morphometry was performed by using statistical parametric mapping. The epsilon4 allele frequency was higher in the AD group (P < 0.001) than the controls but not in the bvFTD group. No differences in demographic or cognitive profiles were observed between epsilon4 allele carriers and noncarriers within any of the diagnostic groups. However, epsilon4 carrier status was associated with more severe brain atrophy in disease-specific regions compared with noncarriers in both AD and bvFTD. AD epsilon4 carriers showed greater atrophy in the bilateral parietal cortex and right hippocampus, and bvFTD epsilon4 carriers demonstrated greater atrophy in the bilateral medial, dorsolateral, and orbital frontal cortex, anterior insula, and cingulate cortex with right predominance. This regional epsilon4 effect is consistent with the hypothesis that apoE may affect the morphologic expression uniquely in different neurodegenerative diseases. The atrophy patterns in epsilon4 carriers may indicate that they are at greater risk for clinical progression.