When a gel swollen with a certain solvent is placed in the bath of another solvent, the gel swells or de-swells depending on the thermodynamic affinity to the gel. Toyotama et al. [Langmuir 22, 1952 (2006)] reported an unusual volume change of chemical gels that cannot be explained by the affinity difference: when a chemical gel saturated with water is immersed in ethylene glycol (EG), although those solvents have almost the same affinity to the polymer, the gel first shrinks and then re-swells and finally takes the same equilibrium volume as the initial. The re-entrant swelling was attributed to different diffusion rates between water and EG (dynamical asymmetry), but the detailed mechanism was not clarified. In this paper, we experimentally show that the characteristic times for the temporal shrinking and subsequent volume relaxation are proportional to the squared system size. This indicates that the phenomenon is governed by diffusive dynamics. According to this observation, we propose a coupled diffusion model explaining the physical mechanism of the re-entrant volume change.