Dark matter with Sommerfeld-enhanced annihilation has been proposed to explain observed cosmic ray positron excesses in the 10 GeV to TeV energy range. We show that the required enhancement implies thermal relic densities that are too small to be all of dark matter. We also show that the dark matter is sufficiently self-interacting that observations of elliptical galactic dark matter halos exclude large Sommerfeld enhancement for light force carriers. Resonant Sommerfeld enhancement does not modify these conclusions, and the astrophysical boosts required to resolve these discrepancies are disfavored, especially when significant self-interactions suppress halo substructure.