Rapid sand filtration is essential at most waterworks that treat anaerobic groundwater. Often the filtration depends on microbiological processes, but the microbial communities of the filters are largely unknown. We determined the prokaryotic community structures of 11 waterworks receiving groundwater from different geological settings by 16S rRNA gene-based 454 pyrosequencing and explored their relationships to filtration technology and raw water chemistry. Most of the variation in microbial diversity observed between different waterworks sand filters could be explained by the geochemistry of the inlet water. In addition, our findings suggested four features of particular interest: (1) Nitrospira dominated over Nitrobacter at all waterworks, suggesting that Nitrospira is a key nitrifying bacterium in groundwater-treating sand filters. (2) Hyphomicrobiaceae species were abundant at all waterworks, where they may be involved in manganese oxidation. (3) Six of 11 waterworks had significant concentrations of methane in their raw water and very high abundance of the methanotrophic Methylococcaceae. (4) The iron-oxidizing bacteria Gallionella was present at all waterworks suggesting that biological iron oxidation is occurring in addition to abiotic iron oxidation. Elucidation of key members of the microbial community in groundwater-treating sand filters has practical potential, for example, when methods are needed to improve filter function.