Multi-modal emotion recognition using EEG and speech signals

Qian Wang; Mou Wang; Yan Yang; Xiaolei Zhang

doi:10.1016/j.compbiomed.2022.105907

Multi-modal emotion recognition using EEG and speech signals

Comput Biol Med. 2022 Oct:149:105907. doi: 10.1016/j.compbiomed.2022.105907. Epub 2022 Jul 22.

Authors

Qian Wang¹, Mou Wang², Yan Yang³, Xiaolei Zhang⁴

Affiliations

¹ Northwestern Polytechnical University, Xi'an, 710072, Shaanxi, China. Electronic address: wangqian203714@mail.nwpu.edu.cn.
² Northwestern Polytechnical University, Xi'an, 710072, Shaanxi, China. Electronic address: wangmou21@mail.nwpu.edu.cn.
³ Northwestern Polytechnical University, Xi'an, 710072, Shaanxi, China. Electronic address: y.yang@nwpu.edu.cn.
⁴ Northwestern Polytechnical University, Xi'an, 710072, Shaanxi, China. Electronic address: xiaolei.zhang@nwpu.edu.cn.

PMID: 36049415
DOI: 10.1016/j.compbiomed.2022.105907

Abstract

Automatic Emotion Recognition (AER) is critical for naturalistic Human-Machine Interactions (HMI). Emotions can be detected through both external behaviors, e.g., tone of voice and internal physiological signals, e.g., electroencephalogram (EEG). In this paper, we first constructed a multi-modal emotion database, named Multi-modal Emotion Database with four modalities (MED4). MED4 consists of synchronously recorded signals of participants' EEG, photoplethysmography, speech and facial images when they were influenced by video stimuli designed to induce happy, sad, angry and neutral emotions. The experiment was performed with 32 participants in two environment conditions, a research lab with natural noises and an anechoic chamber. Four baseline algorithms were developed to verify the database and the performances of AER methods, Identification-vector + Probabilistic Linear Discriminant Analysis (I-vector + PLDA), Temporal Convolutional Network (TCN), Extreme Learning Machine (ELM) and Multi-Layer Perception Network (MLP). Furthermore, two fusion strategies on feature-level and decision-level respectively were designed to utilize both external and internal information of human status. The results showed that EEG signals generate higher accuracy in emotion recognition than that of speech signals (achieving 88.92% in anechoic room and 89.70% in natural noisy room vs 64.67% and 58.92% respectively). Fusion strategies that combine speech and EEG signals can improve overall accuracy of emotion recognition by 25.92% when compared to speech and 1.67% when compared to EEG in anechoic room and 31.74% and 0.96% in natural noisy room. Fusion methods also enhance the robustness of AER in the noisy environment. The MED4 database will be made publicly available, in order to encourage researchers all over the world to develop and validate various advanced methods for AER.

Keywords: Data fusion; EEG emotion recognition; Multi-modal emotion database; Physiological signal; Speech emotion recognition.

Publication types

Research Support, Non-U.S. Gov't

MeSH terms

Algorithms
Discriminant Analysis
Electroencephalography* / methods
Emotions / physiology
Humans
Speech*