Diffractive deep neural networks (D2NNs) have demonstrated their importance in performing various all-optical machine learning tasks such as classification and segmentation. However, current D2NNs can only detect spatial domain intensity information. They cannot solve problems that rely on frequency information, such as laser linewidth compression. We propose a new D2NN architecture that fully exploits frequency domain information. We demonstrate that only frequency domain D2NN (OF-D3NN) can be trained using deep learning algorithms and be successfully integrated into a free-space optical communications system (FSO) for information recovery.