Heart rate recovery (HRR) is a convenient index to assess a cardiovascular autonomic function response to physical exercise. HRR monitoring during daily exercise can be an effective way to verify cardiorespiratory performance. Because HRR varies depending on exercise intensity and resting condition, an exercise condition needs to be acquired for a reliable HRR analysis. This study presents a wearable system for HRR evaluation with automatic labeling of exercise conditions using real-time activity classification. We developed an activity classification algorithm using two features from accelerometer sensor: an acceleration peak and an angle tilt peak. The classification algorithm was applied to a chest-attached wearable device with an embedded electrocardiogram sensor and accelerometer sensors. We classified daily activities such as running, walking, and postural transitions performed under supervised conditions. The wearable device system accurately detected activities with a sensitivity of 99.2 % and posture transitions with a sensitivity of 92 % and specificity of 93.3 % for seven healthy subjects. The proposed wearable system can help monitor HRR during exercise training by labeling the exercise condition simultaneously.