To perform eye or hand movements toward a relevant location, the brain must translate sensory input into motor output. Recent studies revealed segregation between circuits for translating visual information into saccadic or manual movements, but less is known about translation of tactile information into such movements. Using human functional magnetic resonance imaging (fMRI) in a delay paradigm, we factorially crossed sensory modality (vision or touch) and motor effector (eyes or hands) for lateralized movements (gaze shifts to left or right or pressing a left or right button with the corresponding left or right hand located there). We investigated activity in the delay-period between stimulation and response, asking whether the currently relevant side (left or right) during the delay was encoded according to sensory modality, upcoming motor response, or some interactive combination of these. Delay activity mainly reflected the motor response subsequently required. Irrespective of visual or tactile input, we found sustained activity in posterior partial cortex, frontal-eye field, and contralateral visual cortex when subjects would later make an eye movement. For delays prior to manual button-press response, activity increased in contralateral precentral regions, again regardless of stimulated modality. Posterior superior temporal sulcus showed sustained delay activity, irrespective of sensory modality, side, and response type. We conclude that the delay activations reflect translation of sensory signals into effector-specific motor circuits in parietal and frontal cortex (plus an impact on contralateral visual cortex for planned saccades), regardless of cue modality, whereas posterior STS provides a representation that generalizes across both sensory modality and motor effector.