A geographical gradient of height has existed in Japan for approximately 100 years. People in northern Japan tend to be taller than those in southern Japan. The differences in annual temperature and day length between the northern and southern prefectures of Japan have been suggested as possible causes of the height gradient. Although height is well known to be a polygenic trait with high heritability, the genetic contributions to the gradient have not yet been explored. Polygenic score (PS) is calculated by aggregating the effects of genetic variants identified by genome-wide association studies (GWASs) to predict the traits of individual subjects. Here, we calculated the PS of height for 10,840 Japanese individuals from all 47 prefectures in Japan. The median height PS for each prefecture was significantly correlated with the mean height of females and males obtained from another independent Japanese nation-wide height dataset, suggesting genetic contribution to the observed height gradient. We also found that individuals and prefectures genetically closer to continental East Asian ancestry tended to have a higher PS; modern Japanese people are considered to have originated as result of admixture between indigenous Jomon people and immigrants from continental East Asia. Another PS analysis based on the GWAS using only the mainland Japanese was conducted to evaluate the effect of population stratification on PS. The result also supported genetic contribution to height, and indicated that the PS might be affected by a bias due to population stratification even in a relatively homogenous population like Japanese.