Organisms from microbes to humans engage in a variety of social behaviors, which affect fitness in complex, often nonlinear ways. The question of how these behaviors evolve has consequences ranging from antibiotic resistance to human origins. However, evolution with nonlinear social interactions is challenging to model mathematically, especially in combination with spatial, group, and/or kin assortment. We derive a mathematical condition for natural selection with synergistic interactions among any number of individuals. This result applies to populations with arbitrary (but fixed) spatial or network structure, group subdivision, and/or mating patterns. In this condition, nonlinear fitness effects are ascribed to collectives, and weighted by a new measure of collective relatedness. For weak selection, this condition can be systematically evaluated by computing branch lengths of ancestral trees. We apply this condition to pairwise games between diploid relatives, and to dilemmas of collective help or harm among siblings and on spatial networks. Our work provides a rigorous basis for extending the notion of "actor", in the study of social evolution, from individuals to collectives.
Keywords: coalescent theory; collective action; evolutionary dynamics; evolutionary game theory; social evolution.
© The Author(s) 2024. Published by Oxford University Press on behalf of National Academy of Sciences.