Congenital bilateral perisylvian syndrome (CBPS) presents with heterogeneous clinical manifestations such as pseudobulbar palsy, language disorder, variable cognitive deficits, epilepsy, and perisylvian abnormalities (most frequently polymicrogyria) on imaging studies. We investigated the relationship between seizures and extent of gray matter (GM) and white matter (WM) abnormalities using voxel-based morphometry (VBM) of brain magnetic resonance imaging (MRI) as well the association between seizures, structural abnormalities and cognitive ability. In this cross-sectional study, we evaluated 51 healthy volunteers and 18 patients with CBPS with epilepsy (seizure group, n = 7) and without (non-seizure group, n = 11). We used VBM (SPM8/DARTEL) to investigate areas with excess and atrophy of both gray and white matter, comparing groups of patients with controls. Intellectual ability of patients was assessed by the WISC-III or WAIS-III. Both groups with CBPS and the control group were homogeneous with respect to gender (p = 0.07) and age (p = 0.065). Besides perisylvian polymicrogyria, the seizure group exhibited areas with GM and WM reduction including temporal, frontal, parietal and occipital lobes. In contrast, we identified fewer areas with GM and WM reduction in the non-seizure group. The seizure group presented worse intellectual performance (performance IQ and global IQ) than the non-seizure group. The seizure group presented with a more widespread pattern of cortical and sub-cortical abnormalities, as well as worse cognition. Our results suggest that patients with CBPS and epilepsy appear to have widespread neuronal damage that goes beyond the areas with MRI-visible perisylvian polymicrogyria.