Inspired by the robust capability and outstanding performance of convolutional neural networks (CNN) in image classification tasks, CNN-based hyperspectral face recognition methods are worthy of further exploration. However, hyperspectral imaging poses new challenges including high data dimensionality and interference between bands on spectral dimension. High data dimensionality can result in high computational costs. Moreover, not all bands are equally informative and discriminative. The usage of a useless spectral band may even introduce noises and weaken the performance. For the sake of solving those problems, we proposed a novel CNN framework, which adopted a channel-wise attention mechanism and Lasso algorithm to select the optimal spectral bands. The framework is termed as the sparse spectral channel-wise attention-based network (SSCANet) where the SSCA-block focuses on the inter-band channel relationship. Different from other methods which usually select the useful bands manually or in a greedy fashion, SSCA-block can adaptively recalibrate spectral bands by selectively emphasizing informative bands and suppressing less useful ones. Especially, a Lasso constraint strategy can zero out the bands during the training of the network, which can boost the training process by making the weights of bands sparser. Finally, we evaluate the performance of the proposed method in comparison of other state-of-the-art hyperspectral face recognition algorithms on three public datasets HK-PolyU, CMU, and UWA. The experimental results demonstrate that SSCANet based method outperforms the state-of-the-art methods for face recognition on the benchmark.