As human impacts to the environment accelerate, disparities in the distribution of damages between rich and poor nations mount. Globally, environmental change is dramatically affecting the flow of ecosystem services, but the distribution of ecological damages and their driving forces has not been estimated. Here, we conservatively estimate the environmental costs of human activities over 1961-2000 in six major categories (climate change, stratospheric ozone depletion, agricultural intensification and expansion, deforestation, overfishing, and mangrove conversion), quantitatively connecting costs borne by poor, middle-income, and rich nations to specific activities by each of these groups. Adjusting impact valuations for different standards of living across the groups as commonly practiced, we find striking imbalances. Climate change and ozone depletion impacts predicted for low-income nations have been overwhelmingly driven by emissions from the other two groups, a pattern also observed for overfishing damages indirectly driven by the consumption of fishery products. Indeed, through disproportionate emissions of greenhouse gases alone, the rich group may have imposed climate damages on the poor group greater than the latter's current foreign debt. Our analysis provides prima facie evidence for an uneven distribution pattern of damages across income groups. Moreover, our estimates of each group's share in various damaging activities are independent from controversies in environmental valuation methods. In a world increasingly connected ecologically and economically, our analysis is thus an early step toward reframing issues of environmental responsibility, development, and globalization in accordance with ecological costs.