Digital holography based on lensless imaging is a developing method adopted in microscopy and micro-scale measurement. To retrieve complex-amplitude on the sample surface, multiple images are required for common reconstruction methods. A promising single-shot approach points to deep learning, which has been used in lensless imaging but suffering from the unsatisfied generalization ability and stability. Here, we propose and construct a diffraction network (Diff-Net) to connect diffraction images at different distances, which breaks through the limitations of physical devices. The Diff-Net based single-shot holography is robust as there is no practical errors between the multiple images. An iterative complex-amplitude retrieval approach based on light transfer function through the Diff-Net generated multiple images is used for complex-amplitude recovery. This process indicates a hybrid-driven method including both physical model and deep learning, and the experimental results demonstrate that the Diff-Net possesses qualified generalization ability for samples with significantly different morphologies.