How does the brain encode and organize our understanding of the people we know? In this study, participants imagined personally familiar others in a variety of contexts while undergoing fMRI. Using multivoxel pattern analysis, we demonstrated that thinking about familiar others elicits consistent fine-grained patterns of neural activity. Person-specific patterns were distributed across many regions previously associated with social cognition, including medial prefrontal, medial parietal, and lateral temporoparietal cortices, as well as other regions including the anterior and mid-cingulate, insula, and precentral gyrus. Analogous context-specific patterns were observed in medial parietal and superior occipital regions. These results suggest that medial parietal cortex may play a particularly central role in simulating familiar others, as this is the only region to simultaneously represent both person and context information. Moreover, within portions of medial parietal cortex, the degree to which person-specific patterns were typically instated on a given trial predicted subsequent judgments of accuracy and vividness in the mental simulation. This suggests that people may access neural representations in this region to form metacognitive judgments of confidence in their mental simulations. In addition to fine-grained patterns within brain regions, we also observed encoding of both familiar people and contexts in coarse-grained patterns spread across the independently defined social brain network. Finally, we found tentative evidence that several established theories of person perception might explain the relative similarity between person-specific patterns within the social brain network.