Wearable sensors provide a more effective means of activity monitoring and management by recording patients' daily activity data for assessing their daily function and rehabilitation progress, as well as providing a convenient and practical solution for human activity recognition (HAR). However, during the motor rehabilitation of stroke patients, sensors provide vast amounts of high-dimensional data that are large and complex. To enhance the accuracy of activity monitoring and identification, as well as address the limitations of real-time processing, data visualization, and tracking in conventional monitoring approaches, it is essential to perform valid data processing and analysis. This paper combines deep learning models to explore the potential relationships and patterns between data to build an intelligent post-stroke rehabilitation system. This paper proposes a novel framework aimed at accurately recognizing activities performed by stroke patients. Our approach leverages a data fusion mechanism based on multiple sensors to construct a fusion tensor and employs a bidirectional long and short-term memory (BiLSTM) network enhanced with an attention mechanism. This network effectively captures temporal patterns and long-term dependencies within the data, resulting in improved performance for wearable sensor-based activity classification. Furthermore, we introduce an enhanced loss function to optimize the learning process. To assess the performance of the proposed model algorithm, two benchmark datasets were employed. These datasets served as the basis for evaluating and comparing the baseline method as well as other proposed methods. The experimental results clearly demonstrated that the proposed model outperformed the compared methods, indicating its superior performance in activity recognition.