Symbiotic nitrogen (N) fixation is the major N input to many ecosystems. Although temperate forests are commonly N limited, symbiotic N-fixing trees ("N fixers") are rare and decline in abundance as succession proceeds-a challenging paradox that remains unexplained. Understanding demographic processes that underlie N fixers' rarity and successional decline would provide a proximate answer to the paradox. Do N fixers grow slower, die more frequently, or recruit less in temperate forests? We quantified demographic rates of N-fixing and non-fixing trees across succession using U.S. forest inventory data. We used an individual-based model to evaluate the relative contribution of each demographic process to community dynamics. Compared to non-fixers, N fixers had lower growth rates, higher mortality rates, and lower recruitment rates throughout succession. The mortality effect contributed more than the growth effect to N fixers' successional decline. Canopy and understory N fixers experienced these demographic disadvantages, indicating that factors in addition to light limitation likely contribute to N fixers' successional decline. We show that the rarity and successional decline of N-fixing trees in temperate forests is due more to their survival disadvantage than their growth disadvantage, and a recruitment disadvantage might also play a large role.