The use of supervised neural networks is a new approach to solving digital image correlation (DIC) problems, but the existing methods solely adopt the black-box neural network, i.e., the mapping from speckle image pair (reference image and deformed image) to multiple deformation fields (displacement fields and strain fields) is directly established without considering the physical constraints between the fields, causing a low level of accuracy that is even inferior to that of Subset-DIC. In this work, we proposed a deep learning model by introducing strain-displacement relations into a neural network, in which the effect of errors both in displacement and strain are considered in the network training. The back-propagation process of the proposed model is derived, and the solution scheme is implemented by Python. The performance of the proposed model is evaluated by simulation and real DIC experiments, and the results show that adding physical constraints to the neural network can significantly improve prediction accuracy.