Since 1998, tele-rehabilitation has been extensively studied for its potential capacity of saving time and cost for both therapists and patients. However, one gap hindering the deployment of tele-rehabilitation service is the approach to evaluate the outcome after tele-rehabilitation exercises without the presence of professional clinicians. In this paper, we propose an approach to model jerky and jerky-free movement trajectories with hidden Markov models (HMMs). The HMMs are then utilised to identify the jerky characteristics in a motion trajectory, thereby providing the number and amplitude of jerky movements in the specific length of the trajectory. Eventually, the ability of performing functional upper extremity tasks can be evaluated by classifying the motion trajectory into one of the pre-defined ability levels by looking at the number and amplitude of jerky movements. The simulation experiment confirmed that the proposed method is able to correctly classify motion trajectories into various ability levels to a high degree.