We introduce a model in which a particle performs a continuous-time random walk (CTRW) coupled to an environment with Ising dynamics. The particle shows locally varying diffusivity determined by the geometrical properties of the underlying Ising environment, that is, the diffusivity depends on the size of the connected area of spins pointing in the same direction. The model shows anomalous diffusion when the Ising environment is at critical temperature. We show that any finite scale introduced by a temperature different from the critical one, or a finite size of the environment, cause subdiffusion only during a transient time. The characteristic time, at which the system returns to normal diffusion after the subdiffusive plateau depends on the limiting scale and on how close the temperature is to criticality. The system also displays apparent ergodicity breaking at intermediate time, while ergodicity breaking at longer time occurs only under the idealized infinite environment at the critical temperature.