Lightweight ViT Model for Micro-Expression Recognition Enhanced by Transfer Learning

Yanju Liu; Yange Li; Xinhai Yi; Zuojin Hu; Huiyu Zhang; Yanzhong Liu

doi:10.3389/fnbot.2022.922761

Lightweight ViT Model for Micro-Expression Recognition Enhanced by Transfer Learning

Front Neurorobot. 2022 Jun 30:16:922761. doi: 10.3389/fnbot.2022.922761. eCollection 2022.

Authors

Yanju Liu¹, Yange Li², Xinhai Yi², Zuojin Hu¹, Huiyu Zhang², Yanzhong Liu²

Affiliations

¹ School of Mathematics and Information Science, Nanjing Normal University of Special Education, Nanjing, China.
² School of Computer and Control Engineering, Qiqihar University, Qiqihar, China.

Abstract

As opposed to macro-expressions, micro-expressions are subtle and not easily detectable emotional expressions, often containing rich information about mental activities. The practical recognition of micro-expressions is essential in interrogation and healthcare. Neural networks are currently one of the most common approaches to micro-expression recognition. Still, neural networks often increase their complexity when improving accuracy, and overly large neural networks require extremely high hardware requirements for running equipment. In recent years, vision transformers based on self-attentive mechanisms have achieved accuracy in image recognition and classification that is no less than that of neural networks. Still, the drawback is that without the image-specific biases inherent to neural networks, the cost of improving accuracy is an exponential increase in the number of parameters. This approach describes training a facial expression feature extractor by transfer learning and then fine-tuning and optimizing the MobileViT model to perform the micro-expression recognition task. First, the CASME II, SAMM, and SMIC datasets are combined into a compound dataset, and macro-expression samples are extracted from the three macro-expression datasets. Each macro-expression sample and micro-expression sample are pre-processed identically to make them similar. Second, the macro-expression samples were used to train the MobileNetV2 block in MobileViT as a facial expression feature extractor and to save the weights when the accuracy was highest. Finally, some of the hyperparameters of the MobileViT model are determined by grid search and then fed into the micro-expression samples for training. The samples are classified using an SVM classifier. In the experiments, the proposed method obtained an accuracy of 84.27%, and the time to process individual samples was only 35.4 ms. Comparative experiments show that the proposed method is comparable to state-of-the-art methods in terms of accuracy while improving recognition efficiency.

Keywords: computer vision; convolutional neural network; deep learning; micro-expression recognition; vision transformer.