Real-time hand rubbing quality estimation using deep learning enhanced by separation index and feature-based confidence metric

Mohammad Amin Haghpanah; Sina Vali; Amin Mousavi Torkamani; Mehdi Tale Masouleh; Ahmad Kalhor; Ehsan Akhavan Sarraf

doi:10.1016/j.eswa.2023.119588

Real-time hand rubbing quality estimation using deep learning enhanced by separation index and feature-based confidence metric

Expert Syst Appl. 2023 May 15:218:119588. doi: 10.1016/j.eswa.2023.119588. Epub 2023 Jan 21.

Authors

Mohammad Amin Haghpanah¹, Sina Vali², Amin Mousavi Torkamani³, Mehdi Tale Masouleh¹, Ahmad Kalhor¹, Ehsan Akhavan Sarraf⁴

Affiliations

¹ Human and Robot Interaction Laboratory, School of Electrical and Computer Engineering, University of Tehran, Tehran, Iran.
² Human and Robot Interaction Laboratory, University of Tehran, Tehran, Iran.
³ Department of Computational Engineering Lappeenranta University of Technology, Lappeenranta, Finland.
⁴ Tavanresan Company, Tehran, Iran.

Abstract

Hand hygiene plays a crucial role in healthcare environments which can cease infections and diseases from spreading. It is also regarded as the second most effective way to control the transmission of COVID-19. The World Health Organization (WHO) recommends a 12-step guideline for alcohol-based hand rubbing. Compliance with this guideline is vital in order to clean the hands thoroughly. Hence, an automated system can help to improve the quality of this procedure. In this study, a large-scale and diverse dataset for both real and fake hand rubbing motions is collected as the first stage of building a reliable hand hygiene system. In the next stage, various pre-trained networks were analyzed and compared using a swift version of the Separation Index (SI) method. The proposed Swift SI method facilitates choosing the best pre-trained network without fine-tuning them on the whole dataset. Accordingly, the Inception-ResNet architecture achieved the highest SI among Inception, ResNet, Xception, and MobileNet networks. Fine-tuning the Inception-ResNet model led to an accuracy of 98% on the test dataset, which is the highest score in the literature. Therefore, from the proposed approach, a lightweight version of this model with fewer layers but almost the same accuracy is produced and examined. In the final stage, a novel metric, called Feature-Based Confidence (FBC), is devised for estimating the confidence of models in prediction. The proposed confidence measure is able to profoundly differentiate models with similar accuracy and determine the superior one. Based on the metrics results, the Inception-ResNet model is about 2x slower but 5% more confident than its lightweight version. Putting all together, by addressing the real-time application concerns, a Deep Learning based method is offered to qualify the hand rubbing process. The model is also employed in a commercial machine, called DeepHARTS, to estimate the quality of the hand rubbing procedure in different organizations and healthcare environments.

Keywords: Computer vision; Confidence; Deep learning; Hand hygiene; Infection; Machine learning; Real-time classification; Separation index; Transfer learning.