Attention switching between items being stored and manipulated in working memory (WM) is proposed to be an elementary executive function. Experiment 1 reveals a similar attentional limitation within and between verbal and visuospatial WM and identifies a supramodal switching process required for switching between WM items. By using functional magnetic resonance imaging, Experiment 2 investigated brain activation correlates of parametrically varied attention switching within and between these two WM modalities. Attention switching activation was broadly distributed, was quite similar across the three conditions, and, in almost all areas, increased with increasing switching demand, indicating that attention switching recruits and modulates the entire WM network. Dorsolateral prefrontal cortex was implicated in both within- and between-modality attention switching, but no significant activation was found in ventrolateral areas, supporting dorsal-ventral process models of prefrontal organization. A functional dissociation between anterior frontal and dorsolateral prefrontal cortex was found with the former being more activated when switching attention between modalities was required. The data challenge the notion of an anatomically separate attention switching executive function, but suggest that anterior frontal areas are recruited for the additional demand of coordinating the verbal and visuospatial WM slave systems.