Hippocampal subfield atrophy is a prime structural change in the brain, associated with cognitive aging and neurodegenerative diseases such as Alzheimer's disease. Recent developments in genome-wide association studies (GWAS) have identified genetic loci that characterize the risk of hippocampal volume loss based on the processes of normal and abnormal aging. Polygenic risk scores are the genetic proxies mimicking the genetic role of the pre-existing vulnerabilities of the underlying mechanisms influencing these changes. Discriminating the genetic predispositions of hippocampal subfield atrophy between cognitive aging and neurodegenerative diseases will be helpful in understanding the disease etiology. In this study, we evaluated the polygenic risk of Alzheimer's disease (AD PGRS) for hippocampal subfield atrophy in 1,086 individuals (319 cognitively normal (CN), 591 mild cognitively impaired (MCI), and 176 Alzheimer's disease dementia (ADD)). Our results showed a stronger association of AD PGRS effect on the left hemisphere than on the right hemisphere for all the hippocampal subfield volumes in a mixed clinical population (CN+MCI+ADD). The subfields CA1, CA4, hippocampal tail, subiculum, presubiculum, molecular layer, GC-ML-DG, and HATA showed stronger AD PGRS associations with the MCI+ADD group than with the CN group. The subfields CA3, parasubiculum, and fimbria showed moderately higher AD PGRS associations with the MCI+ADD group than with the CN group. Our findings suggest that the eight subfield regions, which were strongly associated with AD PGRS are likely involved in the early stage ADD and a specific focus on the left hemisphere could enhance the early prediction of ADD.