The olfactory system (accessory) implicated in reproductive physiology and behavior in mammals is sexually dimorphic. These brain sex differences present two main characteristics: they are seen in neural circuits related to sexual behavior and sexual physiology and they take one of two opposite morphological patterns (male>female or female>male). The present work reports sex differences in the olfactory system in a large homogeneous sample of men (40) and women (51) using of voxel-based morphology. Gray matter concentration showed sexual dimorphism in several olfactory regions. Women have a higher concentration in the orbitofrontal cortex involving Brodmann's areas 10, 11 and 25 and temporomedial cortex (bilateral hippocampus and right amygdala), as well as their left basal insular cortex. In contrast, men show a higher gray matter concentration in the left entorhinal cortex (Brodmann's area 28), right ventral pallidum, dorsal left insular cortex and a region of the orbitofrontal cortex (Brodmann's area 25). This study supports the hypothesis that the mammalian olfactory system is a sexually dimorphic network and provides a theoretical framework for the morphofunctional approach to sex differences in the human brain.