In recent years there has been a great interest in topological materials and in their fascinating properties. Topological band theory was initially developed for condensed matter systems, but it can be readily applied to arbitrary wave platforms with few modifications. Thus, the topological classification of optical systems is usually regarded as being mathematically equivalent to that of condensed matter systems. Surprisingly, here we find that both the particle-hole symmetry and the dispersive nature of nonreciprocal photonic materials may lead to situations where the usual topological methods break down and the Chern topology becomes ill defined. It is shown that due to the divergence of the density of photonic states in plasmonic systems the gap Chern numbers can be noninteger notwithstanding that the relevant parametric space is compact. In order that the topology of a dispersive photonic crystal is well defined, it is essential to take into account the nonlocal effects in the bulk materials. We propose two different regularization methods to fix the encountered problems. Our results highlight that the regularized topologies may depend critically on the response of the bulk materials for large k.