Apolipoprotein E (APOE) genotype is believed to play a role in the onset of dementia, though less is known about its relationship with non-pathogenic age-related cognitive decline. We assessed whether APOE was a risk factor for cognitive decline among older Taiwanese adults using nationally representative data. General cognition was measured longitudinally over eleven years; domain-specific cognitive assessments of working memory, declarative learning and three aspects of attention (executive function, alerting, and orientation) were performed once. Having at least one risky APOE allele was associated with more rapid longitudinal cognitive decline compared to those with no risky alleles. Some evidence from the cross-sectional analysis of domain-specific cognitive assessments suggested that APOE genotype may be more closely associated with working memory and declarative learning than with attention. Most genetic studies of cognition include only populations of European descent; extension is crucial. This study confirmed the association between APOE genotype and the rate of cognitive decline in a predominantly Han Chinese population. Additional studies on diverse populations are warranted.