In the angular-spectrum method-based computer-generated hologram, the zero-padding method is used to convert circular convolution into linear convolution. However, it will increase the calculation time and memory usage significantly. Therefore, a fast and simple method is proposed to solve the issue of the numerical convolution in the process of hologram generation by using the intermediate angular-spectrum method in this paper. Through replacing numerical Fourier transform by optical Fourier transform in the hologram generation, the calculation speed is approximately 6 times faster than that of the zero-padding method. And due to the scaling factors introduced by the Fourier lens and without the cropping operation, the reconstruction quality of the proposed method is improved significantly compared with the zero-padding method. Moreover, the optical reconstruction system is more compact than the 4-f filter system in the on-axis holographic reconstruction. Both numerical simulations and optical experiments have validated the effectiveness of the proposed method.