We propose a methodology to classify motion of subjects with cerebral palsy based on RGB image sequences and present a new dataset with 2D facial landmark trajectories from RGB images of people with and without disabilities while performing specific types of movements. Depending on these movements, parts of the face can be occluded and we are able to recover the 3D face's shape and its motion based on the Structure from Motion framework. Using the 3D structure and the motion, we propose two different motion descriptors, one is focused on describing the spatial distribution of the motion and the other on the temporal distribution. Finally, we discuss the physical meaning of these descriptors and show that they are very informative about the degree of the subjects' disabilities. Our descriptor can classify people with and without cerebral palsy from 2D image sequences.