Giant negative thermal expansion is achieved in antiperovskite manganese nitrides when the sharp volume change associated with magnetic ordering is broadened by substitution. In this Letter, we address the unique role of the ''magic" element, Ge, for such broadening in Mn3Cu1-xGexN. We present evidence for a local lattice distortion well described by the low-temperature tetragonal (T4) structure of Mn3GeN for a range of x, where the overall structure remains cubic. This structural instability shows a strong correlation with the broadness of the growth of the ordered magnetic moment and, hence, is considered to trigger the broadening of the volume change.