Event-Based Gesture Recognition With Dynamic Background Suppression Using Smartphone Computational Capabilities

Jean-Matthieu Maro; Sio-Hoi Ieng; Ryad Benosman

doi:10.3389/fnins.2020.00275

Event-Based Gesture Recognition With Dynamic Background Suppression Using Smartphone Computational Capabilities

Front Neurosci. 2020 Apr 9:14:275. doi: 10.3389/fnins.2020.00275. eCollection 2020.

Authors

Jean-Matthieu Maro¹, Sio-Hoi Ieng^{1

2}, Ryad Benosman^{1

2

3

4}

Affiliations

¹ Sorbonne Université, INSERM, CNRS, Institut de la Vision, Paris, France.
² CHNO des Quinze-Vingts, INSERM-DGOS CIC 1423, Paris, France.
³ Departments of Ophthalmology/ECE/BioE, University of Pittsburgh, Pittsburgh, PA, United States.
⁴ Department of Computer Science, Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, United States.

Abstract

In this paper, we introduce a framework for dynamic gesture recognition with background suppression operating on the output of a moving event-based camera. The system is developed to operate in real-time using only the computational capabilities of a mobile phone. It introduces a new development around the concept of time-surfaces. It also presents a novel event-based methodology to dynamically remove backgrounds that uses the high temporal resolution properties of event-based cameras. To our knowledge, this is the first Android event-based framework for vision-based recognition of dynamic gestures running on a smartphone without off-board processing. We assess the performances by considering several scenarios in both indoors and outdoors, for static and dynamic conditions, in uncontrolled lighting conditions. We also introduce a new event-based dataset for gesture recognition with static and dynamic backgrounds (made publicly available). The set of gestures has been selected following a clinical trial to allow human-machine interaction for the visually impaired and older adults. We finally report comparisons with prior work that addressed event-based gesture recognition reporting comparable results, without the use of advanced classification techniques nor power greedy hardware.

Keywords: background suppression; dynamic gesture recognition; dynamic vision sensor (DVS); event-based; gesture recognition; mobile device; neuromorphic; smartphone.