Social dilemmas have long been studied formally as cooperation games that pit individual gains against those of the group. In the real world, individuals face an ecology of games where they play many such games simultaneously, often with overlapping co-players. Here, we study an agent-based model of an ecology of public goods games and compare the effectiveness of two institutional mechanisms for promoting cooperation: a simple institution of limited group size (capacity constraints) and a reputational institution based on observed behavior. Reputation is shown to allow much higher relative payoffs for cooperators than do capacity constraints, but only if (1) the rate of reputational information flow is fast enough relative to the rate of social mobility, and (2) cooperators are relatively common in the population. When these conditions are not met, capacity constraints are more effective at protecting the interests of cooperators. Because of the simplicity of the limited-group-size rule, capacity constraints can also generate social organization, which promotes cooperation much more quickly than can reputation. Our results are discussed in terms of both normative prescriptions and evolutionary theory regarding institutions that regulate cooperation. More broadly, the ecology-of-games approach developed here provides an adaptable modeling framework for studying a wide variety of problems in the social sciences.