The epsilon4 allele of the apolipoprotein E (APOE) gene is the major genetic risk factor for Alzheimer's disease (AD), but limited work has suggested that APOE genotype may modulate disease phenotype. Carriers of the epsilon4 allele have been reported to have greater medial temporal lobe (MTL) pathology and poorer memory than noncarriers. Less attention has focused on whether there are domains of cognition and neuroanatomical regions more affected in noncarriers. Further, a major potential confound of prior in vivo studies is the possibility of different rates of clinical misdiagnosis for carriers vs. noncarriers. We compared phenotypic differences in cognition and topography of regional cortical atrophy of epsilon4 carriers (n = 67) vs. noncarriers (n = 24) with mild AD from the Alzheimer's Disease Neuroimaging Initiative, restricted to those with a cerebrospinal fluid (CSF) molecular profile consistent with AD. Between-group comparisons were made for psychometric tests and morphometric measures of cortical thickness and hippocampal volume. Carriers displayed significantly greater impairment on measures of memory retention, whereas noncarriers were more impaired on tests of working memory, executive control, and lexical access. Consistent with this cognitive dissociation, carriers exhibited greater MTL atrophy, whereas noncarriers had greater frontoparietal atrophy. Performance deficits in particular cognitive domains were associated with disproportionate regional brain atrophy within nodes of cortical networks thought to subserve these cognitive processes. These convergent cognitive and neuroanatomic findings in individuals with a CSF molecular profile consistent with AD support the hypothesis that APOE genotype modulates the clinical phenotype of AD through influence on specific large-scale brain networks.