Biological sex differences in brain function and structure are reliably associated with several cortico-subcortical brain regions. While sexual orientation (hetero- versus homosexuality) has been similarly linked to functional differences in several phylogenetically-old brain areas, the research on morphological brain phenotypes associated with sexual orientation is far from conclusive. We examined potential cerebral structural differences linked to sexual orientation in a group of 74 participants, including 37 men (21 homosexual) and 37 women (19 homosexual) using voxel-based morphometry (VBM). Gray matter volumes (GMV) were compared with respect to sexual orientation and biological sex across the entire sample using full factorial designs controlling for total intracranial volume, age, handedness, and education. We observed a significant effect of sexual orientation for the thalamus and precentral gyrus, with more GMV in heterosexual versus homosexual individuals, and for the putamen, with more GMV in homosexual + than heterosexual individuals. We found significant interactions between biological sex and sexual orientation, indicating that the significant effect for the putamen cluster was driven by homosexual women, whereas heterosexual women had increased precentral gyrus GMV. Heterosexual men exhibited more GMV in the thalamus than homosexual men. This study shows that sexual orientation is reflected in brain structure characteristics and that these differ between the sexes. The results emphasize the need to include or control for potential effects of participants' sexual orientation in neuroimaging studies. Furthermore, our findings provide important new insights into the brain morphology underlying sexual orientation and likely have important implications for understanding brain functions and behavior.