The issue of setup errors consisting of translation, rotation and deformation components in head and neck radiotherapy is addressed with a piecewise registration of small independent regions on a portal image to their reference position. These rectangular regions are termed featurelets as they contain relevant anatomical features. The resulting displacement vectors of each featurelet reflect both the center-of-mass (COM), i.e. the rigid, and the non-rigid component of the setup error. The displacement vectors of a series of daily portal images were subjected to a principal component analysis. In addition to the mean, systematic displacement of each featurelet, this analysis yields correlated patterns of anatomical deformations. Hence, the physiological movements of an individual patient can be obtained without a biomechanical model. It is shown that in the presence of setup errors that are due to rotations or deformations a correction by the COM displacement may deteriorate the error of parts of the anatomy further. The featurelet analysis can be used to refine setup correction protocols, tune spatially variable setup margins in treatment planning and optimize patient immobilization devices.