How objects are represented and processed in the brain remains a key issue in cognitive neuroscience. We have developed a conceptual structure account in which category-specific semantic deficits emerge due to differences in the structure and content of concepts rather than from explicit divisions of conceptual knowledge in separate stores. The primary claim is that concepts associated with particular categories (e.g., animals, tools) differ in the number and type of properties and the extent to which these properties are correlated with each other. In this review, we describe recent neuropsychological and neuroimaging studies in which we have extended our theoretical account by incorporating recent claims about the neuroanatomical basis of feature integration and differentiation that arise from research into hierarchical object processing streams in nonhuman primates and humans. A clear picture has emerged in which the human perirhinal cortex and neighbouring anteromedial temporal structures appear to provide the neural infrastructure for making fine-grained discriminations among objects, suggesting that damage within the perirhinal cortex may underlie the emergence of category-specific semantic deficits in brain-damaged patients.