Color rainbow holography can realize color holographic 3D display without speckle noise under white light illumination. However, traditional color rainbow holograms used for high-resolution static color 3D display or near-eye color display are amplitude-type, resulting in low diffraction efficiency due to the presence of conjugate light. In this Letter, a phase-only color rainbow holographic near-eye display is demonstrated. The calculation of a phase-only color rainbow hologram is realized by combining a band-limited diffraction and a bi-directional error diffusion algorithm with high-frequency blazed gratings coded to control longitudinal dispersion. When the wavelength of illumination light is deviated from the designated wavelength of the hologram, only a certain wavefront aberration is caused, but there is no conjugate light. The phase-only color rainbow holographic near-eye display of both a 2D color image and a 3D scene are implemented by optical experiments. It has potential applications in head-mounted 3D augmented reality displays without vergence-accommodation conflict.