We analyze the onset of diffusive hydrodynamics in the one-dimensional hard-rod gas subject to stochastic backscattering. While this perturbation breaks integrability and leads to a crossover from ballistic to diffusive transport, it preserves infinitely many conserved quantities corresponding to even moments of the velocity distribution of the gas. In the limit of small noise, we derive the exact expressions for the diffusion and structure factor matrices, and show that they generically have off diagonal components. We find that the particle density structure factor is non-Gaussian and singular near the origin, with a return probability showing logarithmic deviations from diffusion.