Clustering of magnetic nanoparticles (MNPs) is perhaps the most effective, yet intriguing strategy to enhance T2 relaxivity in magnetic resonance imaging (MRI). However, the underlying mechanism is still not fully understood and the attempts to generalize the classic outersphere theory from single particles to clusters have been found to be inadequate. Here we show that clustering of MNPs enhances local field inhomogeneity due to reduced field symmetry, which can be further elevated by artificially involving iron oxide NPs with heterogeneous geometries in terms of size and shape. The r2 values of iron oxide clusters and Landau-Lifshitz-Gilbert simulations confirmed our hypothesis, indicating that solving magnetic field inhomogeneity may become a powerful way to build correlation between magnetization and T2 relaxivity of MNPs, especially magnetic clusters. This study provides a simple yet distinct mechanism to interpret T2 relaxivity of MNPs, which is crucial to the design of high-performance MRI contrast agents.