In holographic three-dimensional (3D) display, holograms for reconstructing 3D scenes require huge storage space and high transmission bandwidth. Holographic data must be compressed for practical applications. The holograms for 3D display are commonly represented in pure-phase format. Existing hologram compression techniques are generally not designed to handle phase-only holograms. In this paper, we propose a phase-difference-based compression method to compress phase-only holograms for holographic 3D display. Phase-only holograms are decomposed into grayscale images representing the phase distance and binary images containing the sign information of the phase difference. The grayscale images can be better handled by common image compression algorithms since their pixel values are proportional to the phase distance reflecting the distance between complex amplitudes. The binary images are compressed by lossless bi-level image coding. The compressed grayscale images and binary images can be synthesized to recover the phase-only holograms and reconstruct the 3D scenes. The advantages of the proposed method over existing image coding standards are verified by numerical simulations.