One of the central appealing properties of magnetic gels and elastomers is that their elastic moduli can reversibly be adjusted from outside by applying magnetic fields. The impact of the internal magnetic particle distribution on this effect has been outlined and analyzed theoretically. In most cases, however, affine sample deformations are studied and often regular particle arrangements are considered. Here we challenge these two major simplifications by a systematic approach using a minimal dipole-spring model. Starting from different regular lattices, we take into account increasingly randomized structures, until we finally investigate an irregular texture taken from a real experimental sample. On the one hand, we find that the elastic tunability qualitatively depends on the structural properties, here in two spatial dimensions. On the other hand, we demonstrate that the assumption of affine deformations leads to increasingly erroneous results the more realistic the particle distribution becomes. Understanding the consequences of the assumptions made in the modeling process is important on our way to support an improved design of these fascinating materials.