The swelling dynamics of polymer gels are characterized by the (collective) diffusion coefficient D of the polymer network. Here, we measure the temperature dependence of D of polymer gels with controlled homogeneous network structures using dynamic light scattering. An evaluation of the diffusion coefficient at the gelation point D_{gel} and the increase therein as the gelation proceeds ΔD≡D-D_{gel} indicates that ΔD is a linear function of the absolute temperature with a significantly large negative constant term. This feature is formally identical to the recently discovered "negative energy elasticity" [Y. Yoshikawa et al. Phys. Rev. X 11, 011045 (2021)PRXHAE2160-330810.1103/PhysRevX.11.011045], demonstrating a nontrivial similarity between the statics and dynamics of polymer networks.