Mammalian frugivores are critical seed dispersers, but many are under threat of extinction. Futhermore, the impact of past and future defaunation on plant assemblages has yet to be quantified at the global scale. Here, we integrate palm and mammalian frugivore trait and occurrence data and reveal a global positive relationship between fruit size and frugivore body size. Global variation in fruit size is better explained by present-day frugivore assemblages than by Late Pleistocene assemblages, suggesting ecological and evolutionary reorganization after end-Pleistocene extinctions, except in the Neotropics, where some large-fruited palm species may have outlived their main seed dispersers by thousands of years. Our simulations of frugivore extinction over the next 100 years suggest that the impact of defaunation will be highest in the Old World tropics, and an up to 4% assemblage-level decrease in fruit size would be required to maintain the global body size-fruit size relationship. Overall, our results suggest that while some palm species may be able to keep pace with future defaunation through evolutionary changes in fruit size, large-fruited species may be especially vulnerable to continued defaunation.