The source-free Maxwell action is invariant under electric-magnetic duality rotations in arbitrary spacetimes. This leads to a conserved classical Noether charge. We show that this conservation law is broken at the quantum level in the presence of a background classical gravitational field with a nontrivial Chern-Pontryagin invariant, in parallel with the chiral anomaly for massless Dirac fermions. Among the physical consequences, the net polarization of the quantum electromagnetic field is not conserved.