Material-specific memory dysfunction is a feature of mesial temporal lobe epilepsy (mTLE) and has lateralizing potential. We used voxel-based morphometry (VBM) and partial regression analysis of whole-brain tissue class volumes to test if there are correlations between material-specific cognitive dysfunction and localized gray matter loss. In a cohort of 89 patients with mTLE and hippocampal sclerosis (HS), we found correlations between global gray matter and cerebrospinal fluid volume and cognitive test scores in the group with left HS. These findings, however, were poorly anatomically localized; no global changes were detected in the group with right HS. Thus, correlations between gray matter loss and cognitive dysfunction were present and suggested the involvement of widespread neural networks.