Postmenopausal hormone therapy increases mammographic density, a strong breast cancer risk factor, but effects vary across women. We investigated whether the effect of hormone therapy use is modified by polymorphisms in the estrogen receptor (ESR1) and progesterone receptor (PGR) genes in the Dutch Prospect-EPIC and the English EPIC-Norfolk cohorts. Information on hormone therapy use was obtained through questionnaires at recruitment and after 5 years. Blood samples were collected and consecutive mammograms were available through breast cancer screening programs. For 795 hormone therapy users, one mammogram before and a second mammogram during hormone therapy use was included. For 781 never hormone therapy users, mammograms with similar time intervals were included. Mammographic density was assessed using a computer-assisted method. Changes in density were analyzed using linear regression. A statistically significant difference in percentage density change between hormone therapy users and never users was seen in women with the ESR1 PvuII Pp or pp genotype (2.24%; P < 0.01), but not in those with the PP genotype (0.90%; P = 0.47). Similarly, effects of hormone therapy on percentage density were observed in women with the ESR1 XbaI Xx or xx genotype (2.20%; P < 0.01), but not in those with the XX genotype (-0.65%; P = 0.70). Also, effects were seen in women with the PGR +331 GG genotype (2.04%; P < 0.01), but not in those with the GA or AA genotype (0.98%; P = 0.53). The PGR PROGINS polymorphism did not seem to make women more susceptible to the effects of hormone therapy use. In conclusion, our results suggest that specific polymorphisms in the ESR1 and PGR genes may make women more susceptible to the effects of hormone therapy use on mammographic density.