Perception is subjective. Even basic judgments, like those of visual object size, vary substantially between observers and also across the visual field within the same observer. The way in which the visual system determines the size of objects remains unclear, however. We hypothesize that object size is inferred from neuronal population activity in V1 and predict that idiosyncrasies in cortical functional architecture should therefore explain individual differences in size judgments. Here we show results from novel behavioural methods and functional magnetic resonance imaging (fMRI) demonstrating that biases in size perception are correlated with the spatial tuning of neuronal populations in healthy volunteers. To explain this relationship, we formulate a population read-out model that directly links the spatial distribution of V1 representations to our perceptual experience of visual size. Taken together, our results suggest that the individual perception of simple stimuli is warped by idiosyncrasies in visual cortical organization.