Dynamical moduli, such as storage and loss moduli, characterize the viscoelasticity of materials (i.e., time-dependent elasticity) and convey important information about the relaxation processes of glasses and supercooled liquids. A fundamental question is what ultimately determines them in glassy materials. Here, for several model metallic glasses, we demonstrate that both the storage and loss moduli are uniquely determined by the most probable atomic nonaffine displacements, regardless of temperature or frequency. Moreover, the fast-moving atoms (which contribute to dynamical heterogeneity) do not contribute explicitly to the moduli. Our findings provide a physical basis for the origin of viscoelasticity in metallic glasses.