We propose a magneto-optical diffractive deep neural network (MO-D2NN). We simulated several MO-D2NNs, each of which consists of five hidden layers made of a magnetic material that contains 100 × 100 magnetic domains with a domain width of 1 µm and an interlayer distance of 0.7 mm. The networks demonstrate a classification accuracy of > 90% for the MNIST dataset when light intensity is used as the classification measure. Moreover, an accuracy of > 80% is obtained even for a small Faraday rotation angle of π/100 rad when the angle of polarization is used as the classification measure. The MO-D2NN allows the hidden layers to be rewritten, which is not possible with previous implementations of D2NNs.