In grasslands, arbuscular mycorrhizal fungi (AMF) mediate plant diversity; whether AMF increase or decrease diversity depends on the relative mycotrophy in dominant vs. subordinate plants. In this study we investigated whether soil nutrient levels also influence the ability of AMF to mediate plant species coexistence. First, we developed a conceptual model that predicts the influence of AMF on diversity along a soil nutrient gradient for plant communities dominated by mycotrophic and non-mycotrophic species. To test these predictions, we manipulated phosphorus to create a soil nutrient gradient for mesocosm communities composed of native prairie grasses and then compared community properties for mesocosms with and without AMF. We found that, where P was limiting, AMF increased plant diversity and productivity, and also altered community structure; however, at high P, AMF had little influence on aboveground communities. Compositional differences among treatments were due largely to a trade-off in the relative abundance of C3 vs. C4 spes. Our study emphasizes how environmental constraints on mutualisms may govern community- and ecosystem-level properties.