Arbuscular mycorrhizal (AM) fungi are ubiquitous symbionts of higher plants in terrestrial ecosystems, while the occurrence of the AM symbiosis is influenced by a complex set of abiotic and biotic factors. To reveal the regional distribution pattern of AM fungi as driven by multiple environmental factors, and to understand the ecological importance of AM fungi in natural ecosystems, we conducted a field investigation on AM fungal abundance along environmental gradients in the arid and semi-arid grasslands of northern China. In addition to plant parameters recorded in situ, soil samples were collected, and soil chemo-physical and biological parameters were measured in the lab. Statistical analyses were performed to reveal the relative contribution of climatic, edaphic and vegetation factors to AM fungal abundance, especially for extraradical hyphal length density (HLD) in the soil. The results indicated that HLD were positively correlated with mean annual temperature (MAT), soil clay content and soil pH, but negatively correlated with both soil organic carbon (SOC) and soil available N. The multiple regressions and structural equation model showed that MAT was the key positive contributor and soil fertility was the key negative contributor to HLD. Furthermore, both the intraradical AM colonization (IMC) and relative abundance of AM fungi, which was quantified by real-time PCR assay, tended to decrease along the increasing SOC content. With regard to the obvious negative correlation between MAT and SOC in the research area, the positive correlation between MAT and HLD implied that AM fungi could potentially mitigate soil carbon losses especially in infertile soils under global warming. However, direct evidence from long-term experiments is still expected to support the AM fungal contribution to soil carbon pools.