Extension of physical activity recognition with 3D CNN using encrypted multiple sensory data to federated learning based on multi-key homomorphic encryption

Background and objective: The Internet of medical things is enhancing smart healthcare services using physical wearable sensor-based devices connected to the Internet. Machine learning techniques play an important role in the core of these services for remotely consulting patients thanks to the pattern recognition from on-device data, which is transferred to the central servers from local devices. However, transferring personally identifiable information data to servers could become a source for hackers to steal from, manipulate and perform illegal activities. Federated learning is a new branch of machine learning that creates directly training models from on-device data and aggregates these learned models on the servers without centralized data. Another way to protect data confidentiality on computer systems is data encryption. Data encryption transforms data into another form that only users with authority to a decryption key can read. In this work, we propose a novel method enabling preservation of client privacy and protection of client biomedical data from illegal hackers while transmitting through the Internet.

Methods: We propose a method applying 3-dimensional convolutional neural networks for human activity recognition using multiple sensory data. In order to protect the data, we apply the bitwise XOR operator encryption technique. Then, we extend our 3-dimensional convolutional neural network methods to both traditional federated learning and the federated learning based on multi-key homomorphic encryption using the proposed encrypting data.

Results: Based on leave-one-out-cross-validation, the 3-dimensional method obtains an accuracy of 94.6% and of 94.9% (without data encrypting and without federated learning) tested on two different benchmarked datasets, Sport and DaLiAC respectively. Accuracy is decreased slightly to 89.5% (from 94.6% of the baseline) when we use the proposed encrypting data method. However, the encryption-data-based method still has a potential result compared to the state-of-the-art which only uses raw data. In addition, the proposed full federated learning scheme of this work shows that illegal persons who somehow can get the trained model transmitted via networks cannot infer the private result.

Conclusions: This novel method for sensory data representation which translates temporal and frequency bio-signal values to voxel intensities that can encode 3-dimensional activity images. Secondly, the proposed 3-dimensional convolutional neural network methods outperform other deep-learning-based human activity recognition approaches. Finally, extensive experiments show the proposed data-encrypted federated learning approach can achieve feasibility in terms of efficiency in privacy preservation.