Automated sign language detection and classification using reptile search algorithm with hybrid deep learning

Hadeel Alsolai; Leen Alsolai; Fahd N Al-Wesabi; Mahmoud Othman; Mohammed Rizwanullah; Amgad Atta Abdelmageed

doi:10.1016/j.heliyon.2023.e23252

Automated sign language detection and classification using reptile search algorithm with hybrid deep learning

Heliyon. 2023 Dec 8;10(1):e23252. doi: 10.1016/j.heliyon.2023.e23252. eCollection 2024 Jan 15.

Authors

Hadeel Alsolai¹, Leen Alsolai¹, Fahd N Al-Wesabi², Mahmoud Othman³, Mohammed Rizwanullah⁴, Amgad Atta Abdelmageed⁴

Affiliations

¹ Department of Information Systems, College of Computer and Information Sciences, Princess Nourah bint Abdulrahman University, P.O. Box 84428, Riyadh 11671, Saudi Arabia.
² Department of Computer Science, College of Science & Art at Mahayil, King Khalid University, Saudi Arabia.
³ Department of Computer Science, Faculty of Computers and Information Technology, Future University in Egypt New Cairo 11835, Egypt.
⁴ Department of Computer and Self Development, Preparatory Year Deanship, Prince Sattam bin Abdulaziz University, AlKharj, Saudi Arabia.

Abstract

Sign language recognition (SLR) contains the capability to convert sign language gestures into spoken or written language. This technology is helpful for deaf persons or hard of hearing by providing them with a way to interact with people who do not know sign language. It is also be utilized for automatic captioning in live events and videos. There are distinct methods of SLR comprising deep learning (DL), computer vision (CV), and machine learning (ML). One general approach utilises cameras for capturing the signer's hand and body movements and processing the video data for recognizing the gestures. One of challenges with SLR comprises the variability in sign language through various cultures and individuals, the difficulty of certain signs, and require for realtime processing. This study introduces an Automated Sign Language Detection and Classification using Reptile Search Algorithm with Hybrid Deep Learning (SLDC-RSAHDL). The presented SLDC-RSAHDL technique detects and classifies different types of signs using DL and metaheuristic optimizers. In the SLDC-RSAHDL technique, MobileNet feature extractor is utilized to produce feature vectors, and its hyperparameters can be adjusted by manta ray foraging optimization (MRFO) technique. For sign language classification, the SLDC-RSAHDL technique applies HDL model, which incorporates the design of Convolutional Neural Network (CNN) and Long-Short Term Memory (LSTM). At last, the RSA was exploited for the optimal hyperparameter selection of the HDL model, which resulted in an improved detection rate. The experimental result analysis of the SLDC-RSAHDL technique on sign language dataset demonstrates the improved performance of the SLDC-RSAHDL system over other existing DL techniques.

Keywords: Computer vision; Deep learning; Intelligent models; Reptile search algorithm; Sign language.