Similar distribution ranges shared by closely related plant species may have been shaped through different migration histories if those species have differing habitat preference. To test this hypothesis, phylogeographical patterns and population genetic structures were compared between two sister Viola species: Viola eizanensis preferring woodland and V. chaerophylloides var. sieboldiana preferring grassland, both being native to the Japanese Archipelago. Amplified fragment length polymorphism (AFLP) was used for phylogenetic reconstruction, together with Bayesian ancestry analysis, AMOVA, analysis of genetic diversity statistics, and analysis of the relative contribution of each population to total diversity. The results indicated that V. eizanensis had two distinct lineages occurring in the western and eastern part of Japan, but such lineages are not distinct in V. chaerophylloides var. sieboldiana. Both species exhibited the low genetic diversity and high between-population differentiation typical of selfing plants. In V. chaerophylloides var. sieboldiana, one particular population made a significantly higher contribution to the total heterozygosity (H (T)), whereas in V. eizanensis, no population was identified as making a particularly higher contribution to H (T). These findings suggest that V. eizanensis had been isolated in two large glacial refugia, whereas populations of V. chaerophylloides var. sieboldiana were restricted to a single small refuge. Different light requirements between these two closely related species probably caused these differing responses to climatic change during the last ice age.