Recent genetic studies have identified common variation in susceptibility loci that stratify lifetime risks of breast cancer and may inform prevention and screening strategies. However, whether these loci have similar implications for women treated with tamoxifen or raloxifene (SERMs) is unknown. We conducted a matched case-control study of 592 cases who developed breast cancer and 1,171 unaffected women from 32,859 participants on SERM therapy enrolled on NSABP P-1 and P-2 breast cancer prevention trials. We formed a quantitative polygenic risk score (PRS) using genotypes of 75 breast cancer-associated single nucleotide polymorphisms and examined the PRS as a risk factor for breast cancer among women treated with SERMs. The PRS ranged from 3.98 to 7.74, with a one-unit change associated with a 42 % increase in breast cancer (OR = 1.42; P = 0.0002). The PRS had a stronger association with breast cancer among high-risk women with no first-degree family history (OR = 1.62) compared to those with a positive family history (OR = 1.32) (P intx = 0.04). There was also suggestion that PRS was a stronger risk factor for ER-positive (OR = 1.59, P = 0.0002) than ER-negative (OR = 1.05, P = 0.84) breast cancer (P intx = 0.10). Associations did not differ by tamoxifen or raloxifene treatment, age at trial entry, 5-year predicted Gail model risk or other clinical variables. The PRS is a strong risk factor for ER-positive breast cancer in moderate to high-risk individuals treated with either tamoxifen or raloxifene for cancer prevention. These data suggest that common genetic variation informs risk of breast cancer in women receiving SERMs.