Humans are able to categorize complex natural scenes very rapidly and effortlessly, which has led to an assumption that such ultra-rapid categorization is driven by feedforward activation of ventral brain areas. However, recent accounts of visual perception stress the role of recurrent interactions that start rapidly after the activation of V1. To study whether or not recurrent processes play a causal role in categorization, we applied fMRI-guided transcranial magnetic stimulation on early visual cortex (V1/V2) and lateral occipital cortex (LO) while the participants categorized natural images as containing animals or not. The results showed that V1/V2 contributed to categorization speed and to subjective perception during a long activity period before and after the contribution of LO had started. This pattern of results suggests that recurrent interactions in visual cortex between areas along the ventral stream and striate cortex play a causal role in categorization and perception of natural scenes.