Understanding soil microbial communities in agroecosystems has the potential to contribute to the improvement of agricultural productivity and sustainability. Effects of conversion from long-term wheat plantation to Jerusalem artichoke (JA) plantation on soil fungal communities were determined by amplicon sequencing of total fungal ITS regions. Quantitative PCR and PCR-denaturing gradient gel electrophoresis were also used to analyze total fungal and Trichoderma spp. ITS regions and Fusarium spp. Ef1α genes. Results showed that soil organic carbon was higher in the first cropping of JA and Olsen P was lower in the third cropping of JA. Plantation conversion changed soil total fungal and Fusarium but not Trichoderma spp. community structures and compositions. The third cropping of JA had the lowest total fungal community diversity and Fusarium spp. community abundance, but had the highest total fungal and Trichoderma spp. community abundances. The relative abundances of potential fungal pathogens of wheat were higher in the wheat field. Fungal taxa with plant growth promoting, plant pathogen or insect antagonistic potentials were enriched in the first and second cropping of JA. Overall, short-term conversion from wheat to JA plantation changed soil fungal communities, which is related to changes in soil organic carbon and Olsen P contents.