The "irrelevant-change distracting effect" refers to the effect of changes in irrelevant features on the performance of the target feature, which has frequently been used to study information processing in visual working memory (VWM). In the current study, we reported a novel interference effect in VWM: the topological-change interference effect (TCIE). In a series of six experiments, we examined the influence of topological and nontopological changes as irrelevant features on VWM using a color change detection paradigm. The results revealed that only topological changes, although task irrelevant, could produce a significant interference effect. In contrast, nontopological changes did not produce any evident interference effect. Moreover, the TCIE was a stable and lasting effect, regardless of changes in locations, reporting methods, particular stimulus figures, the other salient feature dimensions and delay interval times. Therefore, our results support the notion that topological invariance that defines perceptual objects plays an essential role in maintaining representations in VWM.