Physical activity is one of the key components for elderly in order to be actively ageing. However, it is difficult to differentiate and identify the body movement and actual physical activity using only accelerometer measurements. Therefore, this paper presents an application of a case-based retrieval classification scheme to classify the physical activity of elderly based on pulse rate measurements. Here, a case-based retrieval approach used the features extracted from both time and frequency domain. The evaluation result shows the best accuracy performance while considering the combination of time and frequency domain features. According to the evaluation result while considering the control measurements, the sensitivity, specificity and overall accuracy are achieved as 95%, 96% and 96%, respectively. Considering the test dataset, the system succeeded to identify 13 physical activities out of 16, i.e,. the percentage of the correctness was 81%.