Doping Ni80Fe20 by heavy rare earth atoms alters the magnetic relaxation properties of this material drastically. We show that this effect can be well explained by the slow relaxing impurity mechanism. This process is a consequence of the anisotropy of the on site exchange interaction between the 4f magnetic moments and the conduction band. As expected from this model the magnitude of the damping effect scales with the anisotropy of the exchange interaction and increases by an order of magnitude at low temperatures. In addition, our measurements allow us to determine the relaxation time of the 4f electrons as a function of temperature.