The fractional Fourier transform (FRT) has been used for computing holograms in holographic displays due to its continuity of describing wave diffraction in the near field and far field. In this study, we propose a method to realize a full-color holographic 3D display with combined use of the FRT and the free-space Fresnel diffraction. A slice-based optical configuration and the calculation algorithm of the FRT are proposed for generating phase-only holograms of full-color 3D objects. Sequential phase-only holograms are generated for reducing the speckle noise of reconstructed images by the time-averaging effect. Free-space Fresnel diffraction is used for 3D image reconstruction from the generated holograms. The relationship between the fractional orders of different color channels and the free-space Fresnel diffraction distance is analyzed. Chromatic aberrations caused by different wavelengths of RGB lasers are also compensated. A full-color holographic display system using a reflective phase-only spatial light modulator (SLM) is established. Both the numerical and optical reconstruction results demonstrate the feasibility of the proposed method.