Pre-trained Deep Convolution Neural Network Model With Attention for Speech Emotion Recognition

Hua Zhang; Ruoyun Gou; Jili Shang; Fangyao Shen; Yifan Wu; Guojun Dai

doi:10.3389/fphys.2021.643202

Pre-trained Deep Convolution Neural Network Model With Attention for Speech Emotion Recognition

Front Physiol. 2021 Mar 2:12:643202. doi: 10.3389/fphys.2021.643202. eCollection 2021.

Authors

Hua Zhang^{1

2}, Ruoyun Gou¹, Jili Shang¹, Fangyao Shen¹, Yifan Wu^{1

3}, Guojun Dai¹

Affiliations

¹ School of Computer Science and Technology, HangZhou Dianzi University, Hangzhou, China.
² Key Laboratory of Network Multimedia Technology of Zhejiang Province, Zhejiang University, Hangzhou, China.
³ Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province, HangzhouDianzi University, Hangzhou, China.

Abstract

Speech emotion recognition (SER) is a difficult and challenging task because of the affective variances between different speakers. The performances of SER are extremely reliant on the extracted features from speech signals. To establish an effective features extracting and classification model is still a challenging task. In this paper, we propose a new method for SER based on Deep Convolution Neural Network (DCNN) and Bidirectional Long Short-Term Memory with Attention (BLSTMwA) model (DCNN-BLSTMwA). We first preprocess the speech samples by data enhancement and datasets balancing. Secondly, we extract three-channel of log Mel-spectrograms (static, delta, and delta-delta) as DCNN input. Then the DCNN model pre-trained on ImageNet dataset is applied to generate the segment-level features. We stack these features of a sentence into utterance-level features. Next, we adopt BLSTM to learn the high-level emotional features for temporal summarization, followed by an attention layer which can focus on emotionally relevant features. Finally, the learned high-level emotional features are fed into the Deep Neural Network (DNN) to predict the final emotion. Experiments on EMO-DB and IEMOCAP database obtain the unweighted average recall (UAR) of 87.86 and 68.50%, respectively, which are better than most popular SER methods and demonstrate the effectiveness of our propose method.

Keywords: attention mechanism; deep convolutional neural network; deep neural network; long short-term memory; speech emotion recognition.