Steady-State Visual Evoked Potentials (SSVEP) based Brain-Computer Interface (BCI) has achieved very high information transmission rate (ITR), but its portability and fundamental interactions with the surrounding environment were limited. The combination of Augmented Reality (AR) and BCI is expected to solve these problems. In this paper, we combined AR with the SSVEP-BCI to build a more portable and natural BCI system in Microsoft HoloLens. We designed the AR-BCI system and studied the influence of different algorithms on the system performance. The analysis of SSVEP signals collected in AR environment shows that the extended filter bank canonical correlation analysis was better than task-related component analysis. The average recognition accuracy and ITR obtained by using Electroencephalography (EEG) data of 1s, 1.5s, and 2s length were 87.7%,95.4%, 97.6% and 64.6 bit/min, 62.9 bit/min, 55.6 bit/min, respectively. Compared with the existing AR-BCI studies, the ITR has been greatly improved in this study.