Social categorizations divide people into "us" and "them", often along continuous attributes such as political ideology or skin color. This division results in both positive consequences, such as a sense of community, and negative ones, such as group conflict. Further, individuals in the middle of the spectrum can fall through the cracks of this categorization process and are seen as out-group by individuals on either side of the spectrum, becoming inbetweeners. Here, we propose a quantitative, dynamical-system model that studies the joint influence of cognitive and social processes. We model where two social groups draw the boundaries between "us" and 'them" on a continuous attribute. Our model predicts that both groups tend to draw a more restrictive boundary than the middle of the spectrum. As a result, each group sees the individuals in the middle of the attribute space as an out-group. We test this prediction using U.S. political survey data on how political independents are perceived by registered party members as well as existing experiments on the perception of racially ambiguous faces, and find support.