Evolution of mating systems has become one of the most important research areas in evolutionary biology. Cyrtomium falcatum is a homosporous fern species native to eastern Asia. Two subspecies belonging to a sexual diploid race of C. falcatum are recognized: subsp. littorale and subsp. australe. Subspecies littorale shows intermediate selfing rates, while subsp. australe is an obligate outcrosser. We aimed to evaluate the process of mating system evolution and divergence for the two subspecies using restriction site associated DNA sequencing (RAD-seq). The results showed that subsp. littorale had lower genetic diversity and stronger genetic drift than subsp. australe. Fluctuations in the effective population size over time were evaluated by extended Bayesian skyline plot and Stairway plot analyses, both of which revealed a severe population bottleneck about 20,000 years ago in subsp. littorale. This bottleneck and the subsequent range expansion after the LGM appear to have played an important role in the divergence of the two subspecies and the evolution of selfing in subsp. littorale. These results shed new light on the relationship between mating system evolution and past demographic change in fern species.