Wearable-based human flow experience recognition enhanced by transfer learning methods using emotion data

Muhammad Tausif Irshad; Frédéric Li; Muhammad Adeel Nisar; Xinyu Huang; Martje Buss; Leonie Kloep; Corinna Peifer; Barbara Kozusznik; Anita Pollak; Adrian Pyszka; Olaf Flak; Marcin Grzegorzek

doi:10.1016/j.compbiomed.2023.107489

Wearable-based human flow experience recognition enhanced by transfer learning methods using emotion data

Comput Biol Med. 2023 Sep 22:166:107489. doi: 10.1016/j.compbiomed.2023.107489. Online ahead of print.

Authors

Affiliations

¹ Institute of Medical Informatics, University of Lübeck, Germany. Electronic address: m.irshad@uni-luebeck.de.
² Institute of Medical Informatics, University of Lübeck, Germany. Electronic address: fr.li@uni-luebeck.de.
³ Department of IT, University of the Punjab, Lahore, Pakistan. Electronic address: adeel.nisar@pucit.edu.pk.
⁴ Institute of Medical Informatics, University of Lübeck, Germany. Electronic address: x.huang@uni-luebeck.de.
⁵ Department of Psychology, University of Lübeck, Germany. Electronic address: martje.buss@gmail.com.
⁶ Department of Psychology, University of Lübeck, Germany. Electronic address: l.kloep@uni-luebeck.de.
⁷ Department of Psychology, University of Lübeck, Germany. Electronic address: corinna.peifer@uni-luebeck.de.
⁸ Department of Social Science, Institute of Psychology, University of Silesia in Katowice, Poland. Electronic address: barbara.kozusznik@us.edu.pl.
⁹ Department of Social Science, Institute of Psychology, University of Silesia in Katowice, Poland. Electronic address: anita.pollak@us.edu.pl.
¹⁰ Department of Human Resource Management, College of Management, University of Economics in Katowice, Poland. Electronic address: adrian.pyszka@uekat.pl.
¹¹ Department of Management, Jan Kochanowski University of Kielce, Poland. Electronic address: olaf.flak@ujk.edu.pl.
¹² Institute of Medical Informatics, University of Lübeck, Germany; Department of Knowledge Engineering, University of Economics in Katowice, Poland. Electronic address: marcin.grzegorzek@uni-luebeck.de.

PMID: 37769461
DOI: 10.1016/j.compbiomed.2023.107489

Abstract

Background: Flow experience is a specific positive and affective state that occurs when humans are completely absorbed in an activity and forget everything else. This state can lead to high performance, well-being, and productivity at work. Few studies have been conducted to determine the human flow experience using physiological wearable sensor devices. Other studies rely on self-reported data.

Methods: In this article, we use physiological data collected from 25 subjects with multimodal sensing devices, in particular the Empatica E4 wristband, the Emotiv Epoc X electroencephalography (EEG) headset, and the Biosignalplux RespiBAN - in arithmetic and reading tasks to automatically discriminate between flow and non-flow states using feature engineering and deep feature learning approaches. The most meaningful wearable device for flow detection is determined by comparing the performances of each device. We also investigate the connection between emotions and flow by testing transfer learning techniques involving an emotion recognition-related task on the source domain.

Results: The EEG sensor modalities yielded the best performances with an accuracy of 64.97%, and a macro Averaged F1 (AF1) score of 64.95%. An accuracy of 73.63% and an AF1 score of 72.70% were obtained after fusing all sensor modalities from all devices. Additionally, our proposed transfer learning approach using emotional arousal classification on the DEAP dataset led to an increase in performances with an accuracy of 75.10% and an AF1 score of 74.92%.

Conclusion: The results of this study suggest that effective discrimination between flow and non-flow states is possible with multimodal sensor data. The success of transfer learning using the DEAP emotion dataset as a source domain indicates that emotions and flow are connected, and emotion recognition can be used as a latent task to enhance the performance of flow recognition.

Keywords: Artificial neural network; Deep learning; Flow; Human flow experience; Machine learning; Multimodal sensing; Physiological responses; Transfer learning; Wearable sensors.