The aim of the study is to explore the longitudinal association of dietary acrylamide exposure with cognitive performance in Chinese elderly. The analysis was conducted among 2534 non-smoking elderly men and women based on a prospective study, Mr. and Ms. OS Hong Kong. Dietary acrylamide intake was assessed by food frequency questionnaires with data on local food contamination, derived from the first Hong Kong Total Diet Study. Global cognitive function was assessed by Cantonese version of Mini-Mental State Exam (MMSE) at the baseline and the 4th year of follow-up. Multivariable-adjusted linear and logistic regression models were used to assess the associations of dietary acrylamide with MMSE score changes or risk of poor cognition. The results indicated that among men with MMSE ≥ 18, each one SD increase of acrylamide decreased MMSE score by 7.698% (95%CI: -14.943%, -0.452%; p = 0.037). Logistic regression revealed an increased risk of poor cognition (MMSE ≤ 26) in men with HR of 3.356 (1.064~10.591, p = 0.039). The association became non-significance after further adjustment for telomere length. No significant association was observed in women. Dietary acrylamide exposure was associated with a mild cognitive decline or increased risk of poor cognition over a 4-year period in non-smoking Chinese elderly men.