Blood oxygen level-dependent (BOLD) responses were measured in parts of primary visual cortex that represented unstimulated visual field regions at different distances from a stimulated central target location. The composition of the visual scene varied by the presence or absence of additional peripheral distracter stimuli. Bottom-up effects were assessed by comparing peripheral activity during central stimulation vs. no stimulation. Top-down effects were assessed by comparing active vs. passive conditions. In passive conditions subjects simply watched the central letter stimuli and in active conditions they had to report occurrence of pre-defined targets in a rapid serial letter stream. Onset of the central letter stream enhanced activity in V1 representations of the stimulated region. Within representations of the periphery activation decreased and finally turned into deactivation with increasing distance from the stimulated location. This pattern was most pronounced in the active conditions and during the presence of peripheral stimuli. Active search for a target did not lead to additional enhancement at areas representing the attentional focus but to a stronger deactivation in the vicinity. Suppressed neuronal activity was also found in the non distracter condition suggesting a top-down attention driven effect. Our observations suggest that BOLD signal decreases in primary visual cortex are modulated by bottom-up sensory-driven factors such as the presence of distracters in the visual field as well as by top-down attentional processes.