The goals of this study were to measure if chronic active heavy drinking is associated with brain volume loss in non-treatment seeking men and women, and to assess the effect of positive family history of problem drinking on brain structure in heavy drinkers. Automated image processing was used to analyze high-resolution T1-weighted magnetic resonance images from 49 active heavy drinkers and 49 age- and sex-matched light drinkers, yielding gray matter, white matter and cerebrospinal fluid (CSF) volumes within the frontal, temporal, parietal and occipital lobes. Regional brain volume measures were compared as a function of group, sex and their interaction. Within heavy drinkers, volumes were correlated with measures of alcohol consumption and compared as a function of family history of problem drinking. Deformation morphometry explored localized patterns of atrophy associated with heavy drinking or severity of drinking. We found significant gray matter volume losses, but no white matter losses, in active heavy drinkers compared with light drinkers. Women had greater gray matter and smaller white matter and CSF volumes as a percentage of intracranial vault than men. Within heavy drinkers, smaller gray matter volumes were associated with higher current levels of drinking and older age, while a positive family history of problem drinking was associated with smaller CSF volumes. Community-dwelling heavy drinkers who are not in alcoholism treatment have dose-related gray matter volume losses, and family history of problem drinking ameliorates some structural consequences of heavy drinking.