Androgen deprivation therapy (ADT) is the cornerstone treatment for advanced prostate cancer. Despite frequent responses, the majority of metastatic tumors will progress to castration-resistant prostate cancer. Numerous molecular and genetic perturbations have been described in castration-resistant prostate cancer, which are attributable for gain-of-function changes in the androgen receptor, allowing for cell survival and proliferation with castrate levels of testosterone. The utility of these somatic perturbations, which are selected for in the tumor after ADT, for prognostication of response and response duration in metastatic prostate cancer, is problematic. Here, we discuss recent studies that describe germline polymorphisms that determine the response to ADT. Coding and noncoding germline polymorphisms in genes involved in the androgen pathway affect the response to ADT. These polymorphisms require further study and validation. However, they have the potential to be useful for prognosticating the response to ADT, designing clinical trials for patients who have poor germline prognostic features and designing novel therapies targeted against genes that influence the response to ADT.