To investigate the relationship of Apolipoprotein E (APOE) gene polymorphism to colorectal neoplasia (CRN), we performed a systematic review and meta-analysis. Eligible studies were identified through a systematic literature review from PubMed, EMBASE, and the Science Citation Index up to February 2014. A combined analysis was performed, followed by a subgroup analyses stratified by the study design. We used data collected from 8 prospective studies involving respectively a total of 9243 participants and 4310 CRN cases which including 438 patients with colorectal adenoma (CRA), and 3873 patients with colorectal carcinoma (CRC). The pooled data from this meta-analysis indicated there was no significant association between APOE polymorphism and CRN (ε2: P = 0.51, OR 1.04 95% CI 0.93 to 1.16; ε4: P = 0.72, OR 0.98 95% CI 0.90 to 1.07). Interestingly, subgroup analysis demonstrated there was a significant decreased risk for proximal CRN in patients with APOE ε4 (P = 0.0007, OR 0.52 95% CI 0.35 to 0.76). Data showed no significant association between APOE genotype and overall CRN. However, compared with those carry APOE ε3 alleles, persons with APOE ε4 genotype have significant decreased risk suffering from proximal CRN but not from distal CRN.