The time delays between γ rays of different energies from extragalactic sources have often been used to probe quantum gravity models in which Lorentz symmetry is violated. It has been claimed that these time delays can be explained by or at least put the strongest available constraints on quantum gravity scenarios that cannot be cast within an effective field theory framework, such as the space-time foam, D-brane model. Here we show that this model would predict too many photons in the ultrahigh energy cosmic ray flux to be consistent with observations. The resulting constraints on the space-time foam model are much stronger than limits from time delays and allow for Lorentz violation effects way too small for explaining the observed time delays.