In this Letter, we propose an all-optical diffractive deep neural network modeling method based on nonlinear optical materials. First, the nonlinear optical properties of graphene and zinc selenide (ZnSe) are analyzed. Then the optical limiting effect function corresponding to the saturation absorption coefficient of the nonlinear optical materials is fitted. The optical limiting effect function is taken as the nonlinear activation function of the neural network. Finally, the all-optical diffractive neural network model based on nonlinear materials is established. The numerical simulation results show that the model can effectively improve the nonlinear representation ability of the all-optical diffractive neural network. It provides a theoretical support for the further realization of a photonic artificial intelligence chip based on nonlinear optical materials.