Distributed machine learning (ML) was originally introduced to solve a complex ML problem in a parallel way for more efficient usage of computation resources. In recent years, such learning has been extended to satisfy other objectives, namely, performing learning in situ on the training data at multiple locations and keeping the training datasets private while still allowing sharing of the model. However, these objectives have led to considerable research on the vulnerabilities of distributed learning both in terms of privacy concerns of the training data and the robustness of the learned overall model due to bad or maliciously crafted training data. This article provides a comprehensive survey of various privacy, security, and robustness issues in distributed ML.