Human activity recognition (HAR) is widely applied to many industrial applications. In the context of Industry 4.0, driven by the same demand of machines' self-organizing ability, HAR can also be adopted in elderly healthcare. However, HAR should be adaptive to the application scenarios in elderly healthcare. In this paper, we propose a nonintrusive activity recognition method that can be applied to long-term and unobtrusive monitoring for elderlies. The method is robust to obstruction and nontarget object interference. Skeleton sequence is estimated from RGB images. Based on two activity continuity metrics, an interframe matching algorithm is proposed to filter nontarget objects. In order to make full use of spatial-temporal information, we propose a novel activity encoding method based on the interframe joints distances. A convolutional neural network is used to learn the distinguishing features automatically. A specific data augmentation method is designed to avoid the overfitting problem on small-scale datasets. The experiments are performed on two public activity datasets and a newly released noisy activity dataset (NAD). The NAD contains obstruction, nontarget object interference. The experimental results show that the proposed method achieves the state-of-the-art performance while only using one ordinary camera. The proposed method is robust to a realistic environment.