Measurement of level of consciousness by AVPU scale assessment system based on automated video and speech recognition technology

Dong Hyun Choi; Ki Jeong Hong; Sang Do Shin; Sungwan Kim; Minhwa Chung; Ki Hong Kim; Kyoung Jun Song; Minwoo Cho; Dan Yoon; Jooyoung Lee

doi:10.1016/j.ajem.2023.09.044

Measurement of level of consciousness by AVPU scale assessment system based on automated video and speech recognition technology

Am J Emerg Med. 2023 Dec:74:112-118. doi: 10.1016/j.ajem.2023.09.044. Epub 2023 Sep 27.

Authors

Dong Hyun Choi¹, Ki Jeong Hong², Sang Do Shin³, Sungwan Kim⁴, Minhwa Chung⁵, Ki Hong Kim⁶, Kyoung Jun Song⁷, Minwoo Cho⁸, Dan Yoon⁹, Jooyoung Lee¹⁰

Affiliations

¹ Department of Biomedical Engineering, Seoul National University College of Medicine, Seoul, South Korea; Laboratory of Emergency Medical Services, Seoul National University Hospital Biomedical Research Institute, Seoul, South Korea.
² Department of Emergency Medicine, Seoul National University Hospital, Seoul, South Korea; Laboratory of Emergency Medical Services, Seoul National University Hospital Biomedical Research Institute, Seoul, South Korea; Department of Emergency Medicine, Seoul National University College of Medicine, Seoul, South Korea; Disaster Medicine Research Center, Seoul National University Medical Research Center, Seoul, South Korea. Electronic address: ssberg@snu.ac.kr.
³ Department of Emergency Medicine, Seoul National University Hospital, Seoul, South Korea; Laboratory of Emergency Medical Services, Seoul National University Hospital Biomedical Research Institute, Seoul, South Korea; Department of Emergency Medicine, Seoul National University College of Medicine, Seoul, South Korea; Disaster Medicine Research Center, Seoul National University Medical Research Center, Seoul, South Korea.
⁴ Department of Biomedical Engineering, Seoul National University College of Medicine, Seoul, South Korea; Institute of Bioengineering, Seoul National University, Seoul, South Korea; Artificial Intelligence Institute, Seoul National University, Seoul, South Korea. Electronic address: sungwan@snu.ac.kr.
⁵ Department of Linguistics, Seoul National University College of Humanities, Seoul, South Korea. Electronic address: mchung@snu.ac.kr.
⁶ Department of Emergency Medicine, Seoul National University Hospital, Seoul, South Korea; Laboratory of Emergency Medical Services, Seoul National University Hospital Biomedical Research Institute, Seoul, South Korea; Department of Emergency Medicine, Seoul National University College of Medicine, Seoul, South Korea.
⁷ Department of Emergency Medicine, Seoul National University Boramae Medical Center, Seoul, South Korea; Laboratory of Emergency Medical Services, Seoul National University Hospital Biomedical Research Institute, Seoul, South Korea; Department of Emergency Medicine, Seoul National University College of Medicine, Seoul, South Korea; Disaster Medicine Research Center, Seoul National University Medical Research Center, Seoul, South Korea.
⁸ Transdisciplinary Department of Medicine and Advanced Technology, Seoul National University Hospital, Seoul, South Korea; Artificial Intelligence Institute, Seoul National University, Seoul, South Korea; Medical Big Data Research Center, Seoul National University College of Medicine, Seoul, South Korea.
⁹ Interdisciplinary Program in Bioengineering, Seoul National University Graduate School, Seoul, South Korea; Artificial Intelligence Institute, Seoul National University, Seoul, South Korea.
¹⁰ Department of Linguistics, Seoul National University College of Humanities, Seoul, South Korea. Electronic address: excalibur12@snu.ac.kr.

PMID: 37806172
DOI: 10.1016/j.ajem.2023.09.044

Abstract

Objective: To develop an alert/verbal/painful/unresponsive (AVPU) scale assessment system based on automated video and speech recognition technology (AVPU-AVSR) that can automatically assess a patient's level of consciousness and evaluate its performance through clinical simulation.

Methods: We developed an AVPU-AVSR system with a whole-body camera, face camera, and microphone. The AVPU-AVSR system automatically extracted essential audiovisual features to assess the AVPU score from the recorded video files. Arm movement, pain stimulus, and eyes-open state were extracted using a rule-based approach using landmarks estimated from pre-trained pose and face estimation models. Verbal stimuli were extracted using a pre-trained speech-recognition model. Simulations of a physician examining the consciousness of 12 simulated patients for 16 simulation scenarios (4 for each of "Alert", "Verbal", "Painful", and "Unresponsive") were conducted under the AVPU-AVSR system. The accuracy, sensitivity, and specificity of the AVPU-AVSR system were assessed.

Results: A total of 192 cases with 12 simulated patients were assessed using the AVPU-AVSR system with a multi-class accuracy of 0.95 (95% confidence interval [CI] (0.92-0.98). The sensitivity and specificity (95% CIs) for detecting impaired consciousness were 1.00 (0.97-1.00) and 0.88 (0.75-0.95), respectively. The sensitivity and specificity of each extracted feature ranged from 0.88 to 1.00 and 0.98 to 1.00.

Conclusions: The AVPU-AVSR system showed good accuracy in assessing consciousness levels in a clinical simulation and has the potential to be implemented in clinical practice to automatically assess mental status.

Keywords: AVPU scale; Consciousness; Simulation; Speech recognition; Video recognition.

MeSH terms

Consciousness*
Glasgow Coma Scale
Humans
Pain
Speech
Speech Perception*