Effective task switching is supported by the inhibition of the just executed task, so that potential interference from previously executed tasks is adaptively counteracted. This inhibitory mechanism, named Backward Inhibition (BI), has been inferred from the finding that switching back to a recently executed task (A-B-A task sequence) is harder than switching back to a less recently executed task (C-B-A task sequence). Despite the fact that BI effects do impact performance on everyday life activities, up to now it is still not clear whether the BI represents an amodal and material-independent process or whether it interacts with the task material. To address this issue, a group of individuals with Williams syndrome (WS) characterized by specific difficulties in maintaining and processing visuo-spatial, but not verbal, information, and a mental age- and gender-matched group of typically developing (TD) children were subjected to three task-switching experiments requiring verbal or visuo-spatial material to be processed. Results showed that individuals with WS exhibited a normal BI effect during verbal task-switching, but a clear deficit during visuo-spatial task-switching. Overall, our findings demonstrating that the BI is a material-specific process have important implications for theoretical models of cognitive control and its architecture.