American Sign Language Translation Using Wearable Inertial and Electromyography Sensors for Tracking Hand Movements and Facial Expressions

Yutong Gu; Chao Zheng; Masahiro Todoh; Fusheng Zha

doi:10.3389/fnins.2022.962141

American Sign Language Translation Using Wearable Inertial and Electromyography Sensors for Tracking Hand Movements and Facial Expressions

Front Neurosci. 2022 Jul 19:16:962141. doi: 10.3389/fnins.2022.962141. eCollection 2022.

Authors

Yutong Gu¹, Chao Zheng², Masahiro Todoh³, Fusheng Zha⁴

Affiliations

¹ Graduate School of Engineering, Hokkaido University, Sapporo, Japan.
² Wuhan Second Ship Design and Research Institute, China State Shipbuilding Corporation Limited, Wuhan, China.
³ Faculty of Engineering, Hokkaido University, Sapporo, Japan.
⁴ State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin, China.

Abstract

A sign language translation system can break the communication barrier between hearing-impaired people and others. In this paper, a novel American sign language (ASL) translation method based on wearable sensors was proposed. We leveraged inertial sensors to capture signs and surface electromyography (EMG) sensors to detect facial expressions. We applied a convolutional neural network (CNN) to extract features from input signals. Then, long short-term memory (LSTM) and transformer models were exploited to achieve end-to-end translation from input signals to text sentences. We evaluated two models on 40 ASL sentences strictly following the rules of grammar. Word error rate (WER) and sentence error rate (SER) are utilized as the evaluation standard. The LSTM model can translate sentences in the testing dataset with a 7.74% WER and 9.17% SER. The transformer model performs much better by achieving a 4.22% WER and 4.72% SER. The encouraging results indicate that both models are suitable for sign language translation with high accuracy. With complete motion capture sensors and facial expression recognition methods, the sign language translation system has the potential to recognize more sentences.

Keywords: American sign language; electromyography; inertial measurement units; long short-term memory; transformer.