EEG-based emotion recognition using 4D convolutional recurrent neural network

Fangyao Shen; Guojun Dai; Guang Lin; Jianhai Zhang; Wanzeng Kong; Hong Zeng

doi:10.1007/s11571-020-09634-1

EEG-based emotion recognition using 4D convolutional recurrent neural network

Cogn Neurodyn. 2020 Dec;14(6):815-828. doi: 10.1007/s11571-020-09634-1. Epub 2020 Sep 14.

Authors

Fangyao Shen¹, Guojun Dai^{1

2}, Guang Lin¹, Jianhai Zhang^{1

2}, Wanzeng Kong^{1

2}, Hong Zeng^{1

2}

Affiliations

¹ School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, China.
² Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province, Hangzhou, China.

Abstract

In this paper, we present a novel method, called four-dimensional convolutional recurrent neural network, which integrating frequency, spatial and temporal information of multichannel EEG signals explicitly to improve EEG-based emotion recognition accuracy. First, to maintain these three kinds of information of EEG, we transform the differential entropy features from different channels into 4D structures to train the deep model. Then, we introduce CRNN model, which is combined by convolutional neural network (CNN) and recurrent neural network with long short term memory (LSTM) cell. CNN is used to learn frequency and spatial information from each temporal slice of 4D inputs, and LSTM is used to extract temporal dependence from CNN outputs. The output of the last node of LSTM performs classification. Our model achieves state-of-the-art performance both on SEED and DEAP datasets under intra-subject splitting. The experimental results demonstrate the effectiveness of integrating frequency, spatial and temporal information of EEG for emotion recognition.

Keywords: 4D data; Convolutional recurrent neural network; EEG; Emotion recognition.