Multimodal Emotion Recognition from Art Using Sequential Co-Attention

Tsegaye Misikir Tashu; Sakina Hajiyeva; Tomas Horvath

doi:10.3390/jimaging7080157

Multimodal Emotion Recognition from Art Using Sequential Co-Attention

J Imaging. 2021 Aug 21;7(8):157. doi: 10.3390/jimaging7080157.

Authors

Tsegaye Misikir Tashu^{1

2}, Sakina Hajiyeva¹, Tomas Horvath^{1

3}

Affiliations

¹ Department of Data Science and Engineering (T-Labs), Faculty of Informatics, Eötvös Loránd University, Pázmány Péter Sétány 1/C, 1117 Budapest, Hungary.
² College of Informatics, Kombolcha Institute of Technology, Wollo University, Kombolcha 208, Ethiopia.
³ Faculty of Science Institute of Computer Science, Pavol Jozef Šafárik University, Jesenná 5, 040 01 Košice, Slovakia.

Abstract

In this study, we present a multimodal emotion recognition architecture that uses both feature-level attention (sequential co-attention) and modality attention (weighted modality fusion) to classify emotion in art. The proposed architecture helps the model to focus on learning informative and refined representations for both feature extraction and modality fusion. The resulting system can be used to categorize artworks according to the emotions they evoke; recommend paintings that accentuate or balance a particular mood; search for paintings of a particular style or genre that represents custom content in a custom state of impact. Experimental results on the WikiArt emotion dataset showed the efficiency of the approach proposed and the usefulness of three modalities in emotion recognition.

Keywords: art; attention; emotion analysis; emotions; modality fusion; multimodal.