A visual stimulus activates different sized cortical area depending on eccentricity of the stimulus. Here, our aim is to understand whether the visual field size of a stimulus or cortical size of the corresponding representation determines how strongly it interacts with other stimuli. We measured surround modulation of blood-oxygenation-level-dependent signal and perceived contrast with surrounds that extended either towards the periphery or the fovea from a center stimulus, centered at 6° eccentricity. This design compares the effects of two surrounds which are identical in visual field size, but differ in the sizes of their cortical representations. The surrounds produced equally strong suppression, which suggests that visual field size of the surround determines suppression strength. A modeled population of neuronal responses, in which all the parameters were experimentally fixed, captured the pattern of results both in psychophysics and functional magnetic resonance imaging. Although the fovea-periphery anisotropy affects nearly all aspects of spatial vision, our results suggest that in surround modulation the visual system compensates for it.