Recent studies demonstrate that brain-derived neurotrophic factor (BDNF) might be associated with nicotine addiction, and circulating BDNF is a biomarker of memory and general cognitive function. Moreover, studies suggest that a functional polymorphism of the BDNF Val66Met may mediate hippocampal-dependent cognitive functions. We aimed to explore the relationships between smoking, cognitive performance and BDNF in a normal Chinese Han population. We recruited 628 male healthy subjects, inducing 322 smokers and 306 nonsmokers, and genotyped them the BDNF Val66Met polymorphism. Of these, we assessed 114 smokers and 98 nonsmokers on the repeatable battery for the assessment of neuropsychological status (RBANS), and 103 smokers and 89 nonsmokers on serum BDNF levels. Smokers scored lower than the nonsmokers on RBANS total score (p = 0.002), immediate memory (p = 0.003) and delayed memory (p = 0.021). BDNF levels among the smokers who were Val allele carriers were correlated with the degree of cognitive impairments, especially attention, as well as with the carbon monoxide concentrations. Our findings suggest that smoking is associated with cognitive impairment in a male Chinese Han population. The association between higher BDNF levels and cognitive impairment, mainly attention in smokers appears to be dependent on the BDNF Val66Met polymorphism.