Recently, single-pixel imaging has shown great promise in developing cost-effective imaging systems, where coding and reconstruction are the keys to success. However, it also brings challenges in capturing hyperspectral information accurately and instantly. Many works have attempted to improve reconstruction performance in single-pixel hyperspectral imaging by applying various hand-crafted priors, leading to sub-optimal solutions. In this paper, we present the deep spatial-spectral prior with adaptive dual attention network for single-pixel hyperspectral reconstruction. Specifically, the spindle structure of the parameter sharing method is developed to integrate information across spatial and spectral dimensions of HSI, which can synergistically and efficiently extract global and local prior information of hyperspectral images from both shallow and deep layers. Particularly, a sequential adaptive dual attention block (SADAB), i.e., spatial attention and spectral attention, are devised to adaptively rescale informative features of spatial locations and spectral channels simultaneously, which can effectively boost the reconstruction accuracy. Experiment results on public HSI datasets demonstrate that the proposed method significantly outperforms the state-of-the-art algorithm in terms of reconstruction accuracy and speed.