Activation of systemic immune responses using PD-1 checkpoint inhibitors is an essential approach to cancer therapy. Yet, the extent of benefit relative to risk of immune related adverse events (irAE) varies widely among patients. Here, we study endocrine irAE from 7 clinical trials across 6 cancers where atezolizumab (anti-PD-L1) was combined with chemotherapies and compared to standard of care. We show that atezolizumab-induced thyroid dysfunction is associated with longer survival. We construct a polygenic risk score (PRS) for lifetime risk of hypothyroidism using a GWAS from the UK Biobank and apply this PRS to genetic data collected from 2,616 patients of European ancestry from these trials. Patients with high PRS are at increased risk of atezolizumab-induced thyroid dysfunction and lower risk of death in triple negative breast cancer. Our results indicate that genetic variation associated with thyroid autoimmunity interacts with biological pathways driving the systemic immune response to PD-1 blockade.