Describing the drivers of species loss and of community change are important goals in both conservation and ecology. However, it is difficult to determine whether exploited species decline due to direct effects of harvesting or due to other environmental perturbations brought about by proximity to human populations. Here we quantify differences in species richness of coral reef fish communities along a human population gradient in Papua New Guinea to understand the relative impacts of fishing and environmental perturbation. Using data from published species lists we categorize the reef fishes as either fished or non-fished based on their body size and reports from the published literature. Species diversity for both fished and non-fished groups decreases as the size of the local human population increases, and this relationship is stronger in species that are fished. Additionally, comparison of modern and museum collections show that modern reef communities have proportionally fewer fished species relative to 19th century ones. Together these findings show that the reef fish communities of Papua New Guinea experience multiple anthropogenic stressors and that even at low human population levels targeted species experience population declines across both time and space.