Measuring depression severity based on facial expression and body movement using deep convolutional neural network

Dongdong Liu; Bowen Liu; Tao Lin; Guangya Liu; Guoyu Yang; Dezhen Qi; Ye Qiu; Yuer Lu; Qinmei Yuan; Stella C Shuai; Xiang Li; Ou Liu; Xiangdong Tang; Jianwei Shuai; Yuping Cao; Hai Lin

doi:10.3389/fpsyt.2022.1017064

Measuring depression severity based on facial expression and body movement using deep convolutional neural network

Front Psychiatry. 2022 Dec 21:13:1017064. doi: 10.3389/fpsyt.2022.1017064. eCollection 2022.

Authors

Dongdong Liu¹, Bowen Liu^{2

3}, Tao Lin⁴, Guangya Liu⁵, Guoyu Yang¹, Dezhen Qi¹, Ye Qiu^{1

4}, Yuer Lu^{1

4}, Qinmei Yuan², Stella C Shuai⁶, Xiang Li¹, Ou Liu⁴, Xiangdong Tang⁷, Jianwei Shuai^{1

4

8}, Yuping Cao², Hai Lin⁴

Affiliations

¹ Department of Physics, Fujian Provincial Key Laboratory for Soft Functional Materials Research, Xiamen University, Xiamen, China.
² Department of Psychiatry, National Clinical Research Center for Mental Disorders, The Second Xiangya Hospital of Central South University, Changsha, China.
³ Department of Psychiatry, Baoan Mental Health Center, Shenzhen Baoan Center for Chronic Disease Control, Shenzhen, China.
⁴ Oujiang Laboratory (Zhejiang Lab for Regenerative Medicine, Vision and Brain Health), Wenzhou Key Laboratory of Biophysics, Wenzhou Institute, University of Chinese Academy of Sciences, Wenzhou, Zhejiang, China.
⁵ Integrated Chinese and Western Therapy of Depression Ward, Hunan Brain Hospital, Changsha, China.
⁶ Department of Biological Sciences, Northwestern University, Evanston, IL, United States.
⁷ Sleep Medicine Center, Mental Health Center, Department of Respiratory and Critical Care Medicine, State Key Laboratory of Biotherapy, West China Hospital, Sichuan University, Chengdu, China.
⁸ State Key Laboratory of Cellular Stress Biology, Innovation Center for Cell Signaling Network, National Institute for Data Science in Health and Medicine, Xiamen University, Xiamen, China.

Abstract

Introduction: Real-time evaluations of the severity of depressive symptoms are of great significance for the diagnosis and treatment of patients with major depressive disorder (MDD). In clinical practice, the evaluation approaches are mainly based on psychological scales and doctor-patient interviews, which are time-consuming and labor-intensive. Also, the accuracy of results mainly depends on the subjective judgment of the clinician. With the development of artificial intelligence (AI) technology, more and more machine learning methods are used to diagnose depression by appearance characteristics. Most of the previous research focused on the study of single-modal data; however, in recent years, many studies have shown that multi-modal data has better prediction performance than single-modal data. This study aimed to develop a measurement of depression severity from expression and action features and to assess its validity among the patients with MDD.

Methods: We proposed a multi-modal deep convolutional neural network (CNN) to evaluate the severity of depressive symptoms in real-time, which was based on the detection of patients' facial expression and body movement from videos captured by ordinary cameras. We established behavioral depression degree (BDD) metrics, which combines expression entropy and action entropy to measure the depression severity of MDD patients.

Results: We found that the information extracted from different modes, when integrated in appropriate proportions, can significantly improve the accuracy of the evaluation, which has not been reported in previous studies. This method presented an over 74% Pearson similarity between BDD and self-rating depression scale (SDS), self-rating anxiety scale (SAS), and Hamilton depression scale (HAMD). In addition, we tracked and evaluated the changes of BDD in patients at different stages of a course of treatment and the results obtained were in agreement with the evaluation from the scales.

Discussion: The BDD can effectively measure the current state of patients' depression and its changing trend according to the patient's expression and action features. Our model may provide an automatic auxiliary tool for the diagnosis and treatment of MDD.

Keywords: artificial intelligence; behavioral entropy; deep learning; depression; smart medical.