Vascular contributions to dementia and Alzheimer's disease are increasingly recognized1-6. Recent studies have suggested that breakdown of the blood-brain barrier (BBB) is an early biomarker of human cognitive dysfunction7, including the early clinical stages of Alzheimer's disease5,8-10. The E4 variant of apolipoprotein E (APOE4), the main susceptibility gene for Alzheimer's disease11-14, leads to accelerated breakdown of the BBB and degeneration of brain capillary pericytes15-19, which maintain BBB integrity20-22. It is unclear, however, whether the cerebrovascular effects of APOE4 contribute to cognitive impairment. Here we show that individuals bearing APOE4 (with the ε3/ε4 or ε4/ε4 alleles) are distinguished from those without APOE4 (ε3/ε3) by breakdown of the BBB in the hippocampus and medial temporal lobe. This finding is apparent in cognitively unimpaired APOE4 carriers and more severe in those with cognitive impairment, but is not related to amyloid-β or tau pathology measured in cerebrospinal fluid or by positron emission tomography23. High baseline levels of the BBB pericyte injury biomarker soluble PDGFRβ7,8 in the cerebrospinal fluid predicted future cognitive decline in APOE4 carriers but not in non-carriers, even after controlling for amyloid-β and tau status, and were correlated with increased activity of the BBB-degrading cyclophilin A-matrix metalloproteinase-9 pathway19 in cerebrospinal fluid. Our findings suggest that breakdown of the BBB contributes to APOE4-associated cognitive decline independently of Alzheimer's disease pathology, and might be a therapeutic target in APOE4 carriers.