In silico models based on Deep Neural Networks (DNNs) are promising for predicting activities and properties of new molecules. Unfortunately, their inherent black-box character hinders our understanding, as to which structural features are important for activity. However, this information is crucial for capturing the underlying structure-activity relationships (SARs) to guide further optimization. To address this interpretation gap, "Explainable Artificial Intelligence" (XAI) methods recently became popular. Herein, we apply and compare multiple XAI methods to projects of lead optimization data sets with well-established SARs and available X-ray crystal structures. As we can show, easily understandable and comprehensive interpretations are obtained by combining DNN models with some powerful interpretation methods. In particular, SHAP-based methods are promising for this task. A novel visualization scheme using atom-based heatmaps provides useful insights into the underlying SAR. It is important to note that all interpretations are only meaningful in the context of the underlying models and associated data.