To compute a high-quality computer-generated hologram (CGH) for true 3D real scenes, a huge amount of 3D data must be physically acquired and provided depending on specific devices or 3D rendering techniques. Here, we propose a computational framework for generating a CGH from a single image based on the idea of 2D-to-3D wavefront conversion. We devise a deep view synthesis neural network to synthesize light-field contents from a single image and convert the light-field data to the diffractive wavefront of the hologram using a ray-wave algorithm. The method is able to achieve extremely straightforward 3D CGH generation from hand-accessible 2D image content and outperforms existing real-world-based CGH computation, which inevitably relies on a high-cost depth camera and cumbersome 3D data rendering. We experimentally demonstrate 3D reconstructions of indoor and outdoor scenes from a single image enabled phase-only CGH.