Alzheimer's Disease (AD) is a progressive neurodegenerative disease seen with advancing age. Recent studies have revealed diverse AD-associated cell states, yet when and how they impact the causal chain leading to AD remains unknown. To reconstruct the dynamics of the brain's cellular environment along the disease cascade and to distinguish between AD and aging effects, we built a comprehensive cell atlas of the aged prefrontal cortex from 1.64 million single-nucleus RNA-seq profiles. We associated glial, vascular and neuronal subpopulations with AD-related traits for 424 aging individuals, and aligned them along the disease cascade using causal modeling. We identified two distinct lipid-associated microglial subpopulations, one contributed to amyloid-β proteinopathy while the other mediated the effect of amyloid-β in accelerating tau proteinopathy, as well as an astrocyte subpopulation that mediated the effect of tau on cognitive decline. To model the coordinated dynamics of the entire cellular environment we devised the BEYOND methodology which uncovered two distinct trajectories of brain aging that are defined by distinct sequences of changes in cellular communities. Older individuals are engaged in one of two possible trajectories, each associated with progressive changes in specific cellular communities that end with: (1) AD dementia or (2) alternative brain aging. Thus, we provide a cellular foundation for a new perspective of AD pathophysiology that could inform the development of new therapeutic interventions targeting cellular communities, while designing a different clinical management for those individuals on the path to AD or to alternative brain aging.