Sintering during heterogeneous catalytic reactions is one of the most notorious deactivation channels in catalysts of supported metal nanoparticles. It is therefore critical to understand the effect of support on the sintering behavior. Here, by using in situ aberration-corrected transmission electron microscopy and computational modeling, the atomic-scale dynamic interactions are revealed between Au nanoparticles and various supports. It is found that Au nanoparticles on ceria have a smaller contact angle and are apparently less mobile, especially at surface steps when compared with those on the amorphous silica. Analogous to hydrophilicity, we attribute the origin of mobility of small nanoparticles to metal affinity, which determines the interaction between metal and support material. Ab initio molecular dynamics (AIMD) and machine learning-based deep potential molecular dynamics (DPMD) simulations directly capture a coalescence process on the silica surface and the strong pinning of gold on ceria. The joint experimental and theoretical results on the atomic scale demonstrate the metal affinity of active and inert supports as the key descriptor pertinent to sintering and deactivation of heterogeneous catalysts.