We model the spectral quantum-mechanical purity of heralded single photons from a photon-pair source based on nondegenerate spontaneous four-wave mixing taking the impact of distributed dispersion fluctuations into account. The considered photon-pair-generation scheme utilizes pump-pulse walk-off to produce pure heralded photons and phase matching is achieved through the dispersion properties of distinct spatial modes in a few-mode silica step-index fiber. We show that fiber-core-radius fluctuations in general severely impact the single-photon purity. Furthermore, by optimizing the fiber design we show that generation of single photons with very high spectral purity is feasible even in the presence of large core-radius fluctuations. At the same time, contamination from spontaneous Raman scattering is greatly mitigated by separating the single-photon frequency by more than 32 THz from the pump frequency.