It remains controversial whether APOE E4 polymorphism is related to cognitive function in general population. We aimed to evaluate an association between the APOE E4 genotype and cognitive function, and whether this association may differ by age. Cognitive function was assessed using the Korean version of modified Mini-Mental State Examination (K-mMMSE) in 10,371 Koreans aged 45-74 years in Namwon City. According to the APOE E4 status, all participants were classified as non-carriers, heterozygotes, or homozygotes. Multiple linear and logistic regression models were used to evaluate the association between APOE genotypes and cognition. The frequency of APOE genotypes in the study population was 0.4, 10.1, 1.1, 72.9, 14.7 and 0.8 % for E2E2, E2E3, E2E4, E3E3, E3E4, and E4E4, respectively. Compared to the APOE E4 non-carriers, the heterozygotes and homozygotes showed 1.3 and 7.3 % lower K-mMMSE scores at 65-74 years and 0.8 and 4.6 % higher scores at 45-55 years, respectively. Educational attainment modified the effect of APOE E4 on cognitive function in the 45-54 age group (p for interaction =0.003), showing that the E4 carriers with no-formal education showed significantly higher cognitive function than those with formal education. The present study demonstrates that the effect of APOE E4 on cognitive function depends on age and education.