Oryza rufipogon Griff. (common wild rice; CWR) is the ancestor of Asian cultivated rice (Oryza sativa L.). Investigation of the genetic structure and diversity of CWR in China will provide information about the origin of cultivated rice and the grain quality and yield. In this study, we used 36 simple sequence repeat (SSR) markers to assay 889 accessions, which were highly representative of whole germplasm in China. The analysis revealed a hierarchical genetic structure within CWR. First, CWR has diverged into two ecotypic populations, a south subtropical population (SSP) and a middle subtropical population (MSP), probably owing to natural selection by the different climates. The distribution of specific alleles and haplotypes indicated that Chinese CWR had both indica-like and japonica-like variations; the SSP was an indica-like type, whereas the MSP was more japonica-like. The SSP and MSP further diverged into five (HN, GD-GX1, GX2, FJ and YN) and two (JX-HuN1 and HuN2) geographical populations, respectively. The genetic data suggest the isolation by distance, although water systems also appear to play an important role in the formation of homogenous populations, and occasionally landscape was also involved. The population GD-GX1, which grew widely in Guangdong and Guangxi provinces, was the largest geographical population in China. It had a high level of genetic diversity (GD) and the closest genetic relationship with other inferred populations. The population HN, with the smallest SSR molecular weights and the highest level of GD, may be the most ancestral population.