Annihilating dark matter (DM) has been discussed as a possible source of gamma rays from the galactic center and as a contribution to the extragalactic gamma-ray background. Assuming universality of the density profile of DM halos, we show that it is quite unlikely that DM annihilation is a main constituent of extragalactic gamma-ray background, without exceeding the observed gamma-ray flux from the galactic center. This argument becomes stronger when we include enhancement of the density profiles by supermassive black holes or baryon cooling. The presence of a substructure may loosen the constraint, but only if a very large cross section as well as the rather flat profile are realized.