Ferrogels and magnetic elastomers feature mechanical properties that can be reversibly tuned from outside through magnetic fields. Here we concentrate on the question of how their dynamic response can be adjusted. The influence of three factors on the dynamic behavior is demonstrated using appropriate minimal models: first, the orientational memory imprinted into one class of the materials during their synthesis; second, the structural arrangement of the magnetic particles in the materials; and third, the strength of an external magnetic field. To illustrate the latter point, structural data are extracted from a real experimental sample and analyzed. Understanding how internal structural properties and external influences impact the dominant dynamical properties helps to design materials that optimize the requested behavior.