Given that heart disease can cause abnormal motion dynamics over the cardiac cycle, understanding and quantifying cardiac motion can provide insight for clinicians to aid with diagnosis, therapy planning, and determining prognosis. The goal of this paper is to extract population-specific motion patterns from 3D displacements in order to identify the mean motion in a population, and to describe pathology-specific motion patterns in terms of the spatial and temporal components. Since there are common motion patterns observed in patients with the same condition, extracting these can lead towards a better understanding of the disease. Quantifying cardiac motion at a population level is not a simple task since images can vary widely in terms of image quality, size, resolution, and pose. To overcome this, we analyze the parameters obtained from a cardiac-specific Polyaffine motion-tracking algorithm, which are aligned both spatially and temporally to a common reference space. Once all parameters are aligned, different subjects can be compared and analyzed in the space of Polyaffine transformations by projecting the transformations to a reduced order subspace in which dominant motion patterns in each population can be extracted. Using tensor decomposition, the spatial and temporal aspects can be decoupled in order to study the components individually. The proposed method was validated on healthy volunteers and Tetralogy of Fallot patients according to known spatial and temporal behavior for each population. A key advantage of this method is the ability to regenerate motion sequences from the models, which can be visualized in terms of the full motion.