Many psychiatric disorders are accompanied with sleep abnormalities, having significant influence on emotions which might worsen the disorder conditions. Previous studies discovered that the emotion recognition task with objective physiological signals, such as electroencephalography (EEG) and eye movements, provides a reliable way to figure out the complicated relationship between emotion and sleep. However, both of the emotion and EEG signals are affected by sex. This study aims to investigate how sex differences influence emotion recognition under three different sleep conditions. We firstly developed a four-class emotion recognition task based on various sleep conditions to augment the existing dataset. Then we improved the current state-of-the-art deep-learning model with the attention mechanism. It outperforms the best model with higher accuracy about 91.3% and more stabilization. After that, we compared the results of the male and the female group given by this model. The classification accuracy of happy emotion obviously decreases under sleep deprivation for both males and females, which indicates that sleep deprivation impairs the stimulation of happy emotion. Sleep deprivation also notably weakens the discrimination ability of sad emotion for males while females maintain the same as under common sleep. Our study is instructively beneficial to the real application of emotion recognition in disorder diagnosis.