Person comparison is pervasive in social judgment and human decision making and yet its neural substrate is poorly explored. We measured brain activity when participants compared psychological (intelligence) and physical (height) characteristics of famous people and found activation of medial frontal, orbitofrontal and limbic areas and the temporoparietal junction. This network was largely driven by the psychological comparison, with activity being higher for intelligence than height comparison in several areas in medial prefrontal cortex, suggesting that their activation scales with the demand on person comparison. The person comparison network overlaps strikingly with that commonly described for classic theory of mind tasks. We interpret this overlap as indexing the use of perspective taking common to person comparison and theory of mind.