Many fundamental questions on aging are still unanswered or are under intense debate. These questions are frequently not addressable by examining a single gene or a single pathway, but can best be addressed at the systems level. Here we examined the modular structure of the protein-protein interaction (PPI) networks during fruitfly and human brain aging. In both networks, there are two modules associated with the cellular proliferation to differentiation temporal switch that display opposite aging-related changes in expression. During fly aging, another couple of modules are associated with the oxidative-reductive metabolic temporal switch. These network modules and their relationships demonstrate (1) that aging is largely associated with a small number, instead of many network modules, (2) that some modular changes might be reversible and (3) that genes connecting different modules through PPIs are more likely to affect aging/longevity, a conclusion that is experimentally validated by Caenorhabditis elegans lifespan analysis. Network simulations further suggest that aging might preferentially attack key regulatory nodes that are important for the network stability, implicating a potential molecular basis for the stochastic nature of aging.