Gels are soft elastic materials made of a three-dimensional cross-linked polymer network and featuring both elastic and dissipative responses under external mechanical stimuli. Here we investigate how such gels mediate the organization of embedded magnetic microparticles when driven by an external field. By constructing a continuum theory, we demonstrate that the collective dynamics of the embedded particles result from the delicate balance between magnetic dipole-dipole interactions, thermal fluctuations and elasticity of the polymer network, verified by our experiments. The proposed model could be extended to other soft magnetic composites in order to predict how the elastic interactions mediate the aggregation of the embedded elements, fostering technological implications for multifunctional hydrogel materials.