Rice false smut caused by Villosiclava virens is an economically important disease of grains worldwide. The genetic diversity of 153 isolates from six fields located in Wuhan (WH), Yichang Wangjia (YCW), Yichang Yaohe (YCY), Huanggang (HG), Yangxin (YX), and Jingzhou (JZ) in Hubei province of China were phylogenetically analyzed to evaluate the influence of environments and rice cultivars on the V. virens populations. Isolates (43) from Wuhan were from two rice cultivars, Wanxian 98 and Huajing 952, while most of the other isolates from fields YCW, YCY, HG, YX, and JZ originated from different rice cultivars with different genetic backgrounds. Genetic diversity of isolates was analyzed using random amplified polymorphic DNA (RAPD) and single-nucleotide polymorphisms (SNP). The isolates from the same cultivars in Wuhan tended to group together, indicating that the cultivars had an important impact on the fungal population. The 110 isolates from individual fields tended to cluster according to geographical origin. The values of Nei's gene diversity (H) and Shannon's information index (I) showed that the genetic diversity among isolates was higher between than within geographical populations. Furthermore, mean genetic distance between groups (0.006) was higher than mean genetic distance within groups (0.0048) according to MEGA 5.2. The pairwise population fixation index (FST) values also showed significant genetic differentiation between most populations. Higher genetic similarity of isolates from individual fields but different rice cultivars suggested that the geographical factor played a more important role in the selection of V. virens isolates than rice cultivars. This information could be used to improve the management strategy for rice false smut by adjusting the cultivation measures, such as controlling fertilizer, water, and planting density, in the rice field to change the microenvironment.