Rice landraces, a genetic reservoir for varietal improvement, are developed by farmers through artificial selection during the long-term domestication process. To efficiently conserve, manage, and use such germplasm resources, an understanding of the genetic structure and differentiation of local rice landraces is required. In this study, we analyzed 188 accessions of rice landraces collected from localities across an altitudinal gradient from 425 to 2, 274 m above sea level in Yunnan Province, China using ten target genes and 48 SSR markers. We detected clear differentiation of the rice landraces into indica and japonica groups and further separation of the accessions in each group into two subgroups according to altitude, including a lower altitude subgroup and higher altitude subgroup. The AMOVA results showed significant genetic differentiation among altitude zones at SSRs and most genes, except Os1977 and STS22. We further determined that differentiation among landrace populations followed a model of isolation by altitude, in which gene flow was higher among populations at similar altitude levels than across different altitude levels. Our findings demonstrated that both adaptation to altitude and altitude-dependent gene flow played key roles in the genetic differentiation of rice landraces in Yunnan, China.