Electroencephalography (EEG) signals are often contaminated with various physiological artifacts, seriously affecting the quality of subsequent analysis. Therefore, removing artifacts is an essential step in practice. As of now, deep learning-based EEG denoising methods have exhibited unique advantages over traditional methods. However, they still suffer from the following limitations. The existing structure designs have not fully taken into account the temporal characteristics of artifacts. Meanwhile, the existing training strategies usually ignore the holistic consistency between denoised EEG signals and authentic clean ones. To address these issues, we propose a GAN guided parallel CNN and transformer network, named GCTNet. The generator contains parallel CNN blocks and transformer blocks to respectively capture local and global temporal dependencies. Then, a discriminator is employed to detect and correct the holistic inconsistencies between clean and denoised EEG signals. We evaluate the proposed network on both semi-simulated and real data. Extensive experimental results demonstrate that GCTNet significantly outperforms state-of-the-art networks in various artifact removal tasks, as evidenced by its superior objective evaluation metrics. For example, in the task of removing electromyography artifacts, GCTNet achieves 11.15% reduction in RRMSE and 9.81% improvement in SNR over other methods, highlighting the potential of the proposed method as a promising solution for EEG signals in practical applications.