Hamilton's rule states that cooperation will evolve if the fitness cost to actors is less than the benefit to recipients multiplied by their genetic relatedness. This rule makes many simplifying assumptions, however, and does not accurately describe social evolution in organisms such as microbes where selection is both strong and nonadditive. We derived a generalization of Hamilton's rule and measured its parameters in Myxococcus xanthus bacteria. Nonadditivity made cooperative sporulation remarkably resistant to exploitation by cheater strains. Selection was driven by higher-order moments of population structure, not relatedness. These results provide an empirically testable cooperation principle applicable to both microbes and multicellular organisms and show how nonlinear interactions among cells insulate bacteria against cheaters.