In this review we start from the assumption that, to fully understand cognitive aging, it is important to embrace a holistic view, integrating changes in bodily, brain, and cognitive functions. This broad view can help explain individual differences in aging trajectories and could ultimately enable prevention and remediation strategies. As the title of this review suggests, we claim that there are not only indirect but also direct effects of various organ systems on the brain, creating cascades of phenomena that strongly contribute to age-related cognitive decline. Here we focus primarily on the cerebrovascular system, because of its direct effects on brain health and close connections with the development and progression of Alzheimer's Disease and other types of dementia. We start by reviewing the main cognitive changes that are often observed in normally aging older adults, as well as the brain systems that support them. Second, we provide a brief overview of the cerebrovascular system and its known effects on brain anatomy and function, with a focus on aging. Third, we review genetic and lifestyle risk factors that may affect the cerebrovascular system and ultimately contribute to cognitive decline. Lastly, we discuss this evidence, review limitations, and point out avenues for additional research and clinical intervention.