In supervised dimensionality reduction, tensor representations of images have recently been employed to enhance classification of high dimensional data with small training sets. Previous approaches for handling tensor data have been formulated with tight restrictions on projection directions that, along with convergence issues and the assumption of Gaussian-distributed class data, limit its face-recognition performance. To overcome these problems, we propose a method of rank-one projections with adaptive margins (RPAM) that gives a provably convergent solution for tensor data over a more general class of projections, while accounting for margins between samples of different classes. In contrast to previous margin-based works which determine margin sample pairs within the original high dimensional feature space, RPAM aims instead to maximize the margins defined in the expected lower dimensional feature sub-space by progressive margin refinement after each rank-one projection. In addition to handling tensor data, vector-based variants of RPAM are presented for linear mappings and for nonlinear mappings using kernel tricks. Comprehensive experimental results demonstrate that RPAM brings significant improvement in face recognition over previous subspace learning techniques.