This study was to explore the sex differences in clinical characteristics and brain gray matter volume (GMV) alterations in 29 male patients with major depressive disorder (MDDm), 53 female patients with MDD (MDDf), and in 29 male and 53 female matched healthy controls. Maps of GMV were constructed using magnetic resonance imaging data and compared between groups. We evaluated clinical symptoms using the Hamilton Rating Scale for Depression and obtained a total score and five syndrome scores. A two-factor ANCOVA model was specified using SPM8, with sex and diagnosis as the between-subject factors. We found that: (1) significant GMV increase in the left cerebellum and GMV reduction in the bilateral middle temporal gyrus and left ventral medial prefrontal gyrus occurred selectively in male patients, while the GMV reduction in the left lingual gyrus and dorsal medial prefrontal gyrus occurred selectively in female patients; (2) MDDf may have experienced more severe sleep disturbance than MDDm; and (3) the severity of sleep symptom could be predicted by the sex specific brain structural alterations in depressions. These findings suggest that sex specific anatomical alterations existed in MDD, and these alterations were associated with the clinical symptoms.