Using cortical source estimation techniques based on high-density EEG and fMRI measurements in humans, we measured how a disparity-defined surround influenced the responses to the changing disparity of a central disk within five visual ROIs: V1, V4, lateral occipital complex (LOC), hMT+, and V3A. The responses in the V1 ROI were not consistently affected either by changes in the characteristics of the surround (correlated or uncorrelated) or by its disparity value, consistent with V1 being responsive only to absolute, not relative, disparity. Correlation in the surround increased the responses in the V4, LOC, and hMT+ ROIs over those measured with the uncorrelated surround. Thus, these extrastriate areas contain neurons that are sensitive to disparity differences. However, their evoked responses did not vary systematically with the surround disparity. Responses in the V3A ROI, in contrast, were increased by correlation in the surround and varied with its disparity. We modeled these V3A responses as attributable to a gain modulation of the absolute disparity response, where the gain amplitude is proportional to the center-surround disparity difference. An additional experiment identified a nonlinear center-surround interaction in V3A that facilitates the responses when center and surround are misaligned but suppresses it when they share the same disparity plane.