The evergreen broadleaved forests (EBLFs) of Southwest China have a long-term stable environment and support a diverse flora, thus forming a global biodiversity hotspot. However, the key drivers that have shaped the genetic diversity patterns of species in these EBLFs are not well understood. Quercus delavayi, Q. schottkyana, and Q. kerrii are sympatric oak species with different seed biological traits that are typical for these EBLFs. This study combined multilocus phylogeography and ecological niche modeling to screen 33 Q. delavayi populations. Their population genetic structure was inferred in comparison with previous studies on Q. schottkyana and Q. kerrii. The seed germination traits of all three species were also investigated. cpDNAs showed a significant phylogeographic structure in Q. delavayi, which was not detected in Q. schottkyana or Q. kerrii. Quercus delavayi, Q. kerrii, and Q. schottkyana exhibited different pollen-to-seed migration ratios (r = 219, 117, and 22, respectively), which are linked to the germination schedules of acorns. The distributions of Q. delavayi and Q. schottkyana remained long-term stable since the last glacial maximum (LGM) with a similar nSSR genetic gradient change along latitude. Instead, Q. kerrii experienced a prominent range expansion since the LGM with genetic diversification between the East and the West of the Tanaka line due to environmental heterogeneity. These results identify seed traits and environmental heterogeneity as two key drivers that shape the population genetic structure of EBLF trees in Southwest China. These should be considered in regional forestry conservation and management.