Universally, female skin color is lighter than male skin color, irrespective of geographical location. This difference is a distinctive and universal adaptive pattern that emerges after puberty. We address whether this sexual dimorphism is cognitively and culturally represented to ground gender. To this end, we examine a non-Western, non-industrialized population, namely the Wichí (Salta, Argentina) and a Western industrialized population (Spain). The two cultural populations included both adults and prepubescent children. Across two experiments, we utilized a novel task with children and adults who had to make a choice for a female (male) target person between two identical objects that differed only in terms of their brightness. The results in both experiments revealed that the children from the two cultural communities choose a lighter colored object for the female target and a darker version of the same object for the male target. This pattern held across cultures irrespective of the age of participants, except for the male Wichí participants. We discuss how sexual dimorphism in skin color contributes to a universal grounding of the gender category, and advance possible explanations as to why Wichi males did not consistently link gender and brightness.