A physics-based deep learning (DL) method termed Phynet is proposed for modeling the nonlinear pulse propagation in optical fibers totally independent of the ground truth. The presented Phynet is a combination of a handcrafted neural network and the nonlinear Schrödinger physics model. In particular, Phynet is optimized through physics loss generated by the interaction between the network and the physical model rather than the supervised loss. The inverse pulse propagation problem is leveraged to exemplify the performance of Phynet when in comparison to the typical DL method under the same structure and datasets. The results demonstrate that Phynet is able to precisely restore the initial pulse profiles with varied initial widths and powers, while revealing a similar prediction accuracy compared with the typical DL method. The proposed Phynet method can be expected to break the severe bottleneck of the traditional DL method in terms of relying on abundant labeled data during the training phase, which thus brings new insight for modeling and predicting the nonlinear dynamics of the fibers.