Class-dependent and cross-modal memory network considering sentimental features for video-based captioning

Haitao Xiong; Yuchen Zhou; Jiaming Liu; Yuanyuan Cai

doi:10.3389/fpsyg.2023.1124369

Class-dependent and cross-modal memory network considering sentimental features for video-based captioning

Front Psychol. 2023 Feb 15:14:1124369. doi: 10.3389/fpsyg.2023.1124369. eCollection 2023.

Authors

Haitao Xiong¹, Yuchen Zhou¹, Jiaming Liu¹, Yuanyuan Cai^{2

3}

Affiliations

¹ School of International Economics and Management, Beijing Technology and Business University, Beijing, China.
² National Engineering Research Centre for Agri-Product Quality Traceability, Beijing Technology and Business University, Beijing, China.
³ School of E-Business and Logistics, Beijing Technology and Business University, Beijing, China.

Abstract

The video-based commonsense captioning task aims to add multiple commonsense descriptions to video captions to understand video content better. This paper aims to consider the importance of cross-modal mapping. We propose a combined framework called Class-dependent and Cross-modal Memory Network considering SENtimental features (CCMN-SEN) for Video-based Captioning to enhance commonsense caption generation. Firstly, we develop class-dependent memory for recording the alignment between video features and text. It only allows cross-modal interactions and generation on cross-modal matrices that share the same labels. Then, to understand the sentiments conveyed in the videos and generate accurate captions, we add sentiment features to facilitate commonsense caption generation. Experiment results demonstrate that our proposed CCMN-SEN significantly outperforms the state-of-the-art methods. These results have practical significance for understanding video content better.

Keywords: class-dependent memory; commonsense caption; cross-modal mapping; cross-modal matrices; cross-modal memory network; sentimental features.