There are mixed reports on structural neuroimaging correlates of aggression in schizophrenia with weak evidence due to cohort overlaps and lack of replications. To our knowledge, no study examined volumetric neuroimaging correlates of aggression in early stages of psychosis. An agitated-aggressive syndrome is present in at-risk mental state (ARMS) and in first-episode psychosis (FEP) - it is unclear whether this syndrome is associated with structural brain abnormalities in early stages of psychosis. Using three-dimensional magnetic resonance imaging and a whole brain voxel-based morphometry approach, we examined 56 ARMS patients, 55 FEP patients and 25 healthy controls. We operationalized aggression using the Excited Component of the Brief Psychiatric Rating Scale (BPRS-EC) and dichotomized our patient group by median split into "BPRS-EC high" (n = 49) and "BPRS-EC low" groups (n = 62). The "BPRS-EC high" group had significantly smaller left lingual gyrus volume than HC. This finding was not present in the "BPRS-EC low" group. In addition, grey matter volume in the left lingual gyrus showed a negative linear correlation with BPRS-EC over all subjects (ρ = -0.318; p = 0.0001) and in the patient group (ρ = -0.202; p = 0.033). These findings provide first hints on structural brain abnormalities associated with an agitated-aggressive syndrome in ARMS and FEP patients.