Climate relicts hold considerable importance because they have resulted from numerous historical changes. However, there are major interspecific variations among the ways by which they survived climate changes. Therefore, investigating the factors and timing that affected population demographics can expand our understanding of how climate relicts responded to historical environmental changes. Here, we examined herbaceous hydrangeas of genus Deinanthe in East Asia, which show limited distributions and a remarkable disjunction between Japan and central China. Chloroplast genome and restriction site-associated DNA sequencing revealed that speciation event occurred in the late Miocene (ca. 7-9 Mya) in response to global climate change. Two lineages apparently remained not branched until the middle Quaternary, and afterwards started to diverge to regional population groups. The narrow endemic species in central China showed lower genetic diversity (He = 0.082), as its population size rapidly decreased during the Holocene due to isolation in montane refugia. Insular populations in the three Japanese islands (He = 0.137-0.160) showed a genetic structure that was inconsistent with sea barriers, indicating that it was shaped in the glacial period when its range retreated to coastal refugia on the exposed sea floor. Demographic modelling by stairway-plot analysis reconstructed variable responses of Japanese populations: some experienced glacial bottlenecks in refugial isolation, while post-glacial range expansion seemingly exerted founder effects on other populations. Overall, this study demonstrated the involvement of not just one, but multiple factors, such as the interplay between climate changes, geography, and other population-specific factors, that determine the demographics of climate relicts.