Emotion recognition from EEG based on multi-task learning with capsule network and attention mechanism

Chang Li; Bin Wang; Silin Zhang; Yu Liu; Rencheng Song; Juan Cheng; Xun Chen

doi:10.1016/j.compbiomed.2022.105303

Emotion recognition from EEG based on multi-task learning with capsule network and attention mechanism

Comput Biol Med. 2022 Apr:143:105303. doi: 10.1016/j.compbiomed.2022.105303. Epub 2022 Feb 19.

Authors

Chang Li¹, Bin Wang², Silin Zhang³, Yu Liu⁴, Rencheng Song⁵, Juan Cheng⁶, Xun Chen⁷

Affiliations

¹ Department of Biomedical Engineering, Hefei University of Technology, Hefei, 230009, China; Anhui Province Key Laboratory of Measuring Theory and Precision Instrument, Hefei University of Technology, Hefei, 230009, China. Electronic address: changli@hfut.edu.cn.
² Department of Biomedical Engineering, Hefei University of Technology, Hefei, 230009, China. Electronic address: binwang@mail.hfut.edu.cn.
³ Reproductive Medical Center, Renmin Hospital of Wuhan University, Wuhan, 430060, China. Electronic address: rm003305@whu.edu.cn.
⁴ Department of Biomedical Engineering, Hefei University of Technology, Hefei, 230009, China. Electronic address: yuliu@hfut.edu.cn.
⁵ Department of Biomedical Engineering, Hefei University of Technology, Hefei, 230009, China. Electronic address: rcsong@hfut.edu.cn.
⁶ Department of Biomedical Engineering, Hefei University of Technology, Hefei, 230009, China. Electronic address: chengjuan@hfut.edu.cn.
⁷ Department of Neurosurgery, The First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, 230001, Anhui, China; Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, 230026, China. Electronic address: xunchen@ustc.edu.cn.

PMID: 35217341
DOI: 10.1016/j.compbiomed.2022.105303

Abstract

Deep learning (DL) technologies have recently shown great potential in emotion recognition based on electroencephalography (EEG). However, existing DL-based EEG emotion recognition methods are built on single-task learning, i.e., learning arousal, valence, and dominance individually, which may ignore the complementary information of different tasks. In addition, single-task learning involves a new round of training every time a new task appears, which is time consuming. To this end, we propose a novel method for EEG-based emotion recognition based on multi-task learning with capsule network (CapsNet) and attention mechanism. First, multi-task learning can learn multiple tasks simultaneously while exploiting commonalities and differences across tasks, it can also obtain more data from different tasks, which can improve generalization and robustness. Second, the innovative structure of the CapsNet enables it to effectively characterize the intrinsic relationship among various EEG channels. Finally, the attention mechanism can change the weight of different channels to extract important information. In the DEAP dataset, the average accuracy reached 97.25%, 97.41%, and 98.35% on arousal, valence, and dominance, respectively. In the DREAMER dataset, average accuracy reached 94.96%, 95.54%, and 95.52% on arousal, valence, and dominance, respectively. Experimental results demonstrate the efficiency of the proposed method for EEG emotion recognition.

Keywords: Capsule network; Deep learning; Electroencephalogram; Emotion recognition; Multi-task learning.