Magnetic gels and elastomers are promising candidates to construct reversibly excitable soft actuators, triggered from outside by magnetic fields. These magnetic fields induce or alter the magnetic interactions between discrete rigid particles embedded in a soft elastic polymeric matrix, leading to overall deformations. It is a major challenge in theory to correctly predict from the discrete particle configuration the type of deformation resulting for a finite-sized system. Considering an elastic sphere, we here present such an approach. The method is in principle exact, at least within the framework of linear elasticity theory and for large enough interparticle distances. Different particle arrangements are considered. We find, for instance, that regular simple cubic configurations show elongation of the sphere along the magnetization if oriented along a face or space diagonal of the cubic unit cell. Contrariwise, with the magnetization along the edge of the cubic unit cell, they contract. The opposite is true in this geometry for body- and face-centered configurations. Remarkably, for the latter configurations but the magnetization along a face or space diagonal of the unit cell, contraction was observed to revert to expansion with decreasing Poisson ratio of the elastic material. Randomized configurations were considered as well. They show a tendency of elongating the sphere along the magnetization, which is more pronounced for compressible systems. Our results can be tested against actual experiments for spherical samples. Moreover, our approach shall support the search of optimal particle distributions for a maximized effect of actuation.