Phase retrieval (PR), a long-established challenge for recovering a complex-valued signal from its Fourier intensity-only measurements, has attracted considerable attention due to its widespread applications in optical imaging. Recently, deep learning-based approaches were developed and allowed single-shot PR. However, due to the substantial disparity between the input and output domains of the PR problems, the performance of these approaches using vanilla deep neural networks (DNN) still has much room to improve. To increase the reconstruction accuracy, physics-informed approaches were suggested to incorporate the Fourier intensity measurements into an iterative estimation procedure. Since the approach is iterative, they require a lengthy computation process, and the accuracy is still not satisfactory for images with complex structures. Besides, many of these approaches work on simulation data that ignore some common problems such as saturation and quantization errors in practical optical PR systems. In this paper, a novel physics-driven multi-scale DNN structure dubbed PPRNet is proposed. Similar to other deep learning-based PR methods, PPRNet requires only a single Fourier intensity measurement. It is physics-driven that the network is guided to follow the Fourier intensity measurement at different scales to enhance the reconstruction accuracy. PPRNet has a feedforward structure and can be end-to-end trained. Thus, it is much faster and more accurate than the traditional physics-driven PR approaches. Extensive simulations and experiments on an optical platform were conducted. The results demonstrate the superiority and practicality of the proposed PPRNet over the traditional learning-based PR methods.