In this paper, we present a novel high-frequency facial feature and a high-frequency based sparse representation classification to tackle single sample face recognition (SSFR) under varying illumination. Firstly, we propose the assumption that QRCP bases can represent intrinsic face surface features with different frequencies, and their corresponding energy coefficients describe illumination intensities. Based on this assumption, we take QRCP bases with corresponding weighting coefficients (i.e. the major components of energy coefficients) to develop the high-frequency facial feature of the face image, which is named as QRCP-face. The normalized QRCP-face (NQRCPface) is constructed to further constraint illumination effects by normalizing the weighting coefficients of QRCP-face. Moreover, we propose the adaptive QRCP-face (AQRCP-face) that assigns a special parameter to NQRCP-face via the illumination level estimated by the weighting coefficients. Secondly, we consider that the differences of pixel images cannot model the intraclass variations of generic faces with illumination variations, and the specific identification information of the generic face is redundant for the current SSFR with generic learning. To tackle above two issues, we develop a general high-frequency based sparse representation (GHSP) model. Two practical approaches separated high-frequency based sparse representation (SHSP) and unified high-frequency based sparse representation (UHSP) are developed. Finally, the performances of the proposed methods are verified on the Extended Yale B, CMU PIE, AR, LFW and our self-built Driver face databases. The experimental results indicate that the proposed methods outperform previous approaches for SSFR under varying illumination.