In Alzheimer's disease (AD), atrophy negatively impacts cognition while in healthy adults, inverse relationships between brain volume and cognition may occur. We investigated correlations between gray matter volume and cognition in elderly controls, AD and mild cognitive impairment (MCI) patients with memory and executive deficits. AD demonstrated substantial loss in temporal, parietal and frontal regions while MCI exhibited moderate volume loss in temporal and frontal regions. In controls, memory and executive function were negatively correlated with frontal regions, while in AD, memory was positively correlated with temporal and frontal gyri, and executive function with frontal regions. The combination of the two patterns may explain the lack of correlations in MCI. Developmental versus pathological contributions to these relationships are discussed.