The human neural correlates of social rejection have attracted significant research interest, but remain subject to vigorous debate. Specifically, it has been proposed that a matrix of brain regions overlapping with the classical pain matrix, and including the dorsal anterior cingulate cortex (dACC) and the anterior insular cortex (AI) is critical for processing of social rejection. The present study expands on this conceptualization, by showing that these areas are involved in processing of self-relevant social evaluation, irrespective of valence. Forty healthy adolescents (N = 20 females) were tested in a magnetic resonance imaging (MRI) scanner. We used a novel paradigm that balanced participants' experience of rejection and acceptance. In addition, the paradigm also controlled for whether the social judgment was towards the participants or towards other fictitious players. By creating a "self" and "other" distinction, we show that right AI and dACC are involved in processing the salience of being judged by others, irrespective of the quality of this judgment. This finding supports the idea that these regions are not specific to social rejection or even to pain or metaphorically painful experiences, but activate to self-relevant, highly salient information.