The geological record of marine animal biodiversity reflects the interplay between changing rates of speciation versus extinction. Compared to mass extinctions, background extinctions have received little attention. To disentangle the different contributions of global climate state, continental configuration, and atmospheric oxygen concentration (pO2) to variations in background extinction rates, we drive an animal physiological model with the environmental outputs from an Earth system model across intervals spanning the past 541 million years. We find that climate and continental configuration combined to make extinction susceptibility an order of magnitude higher during the Early Paleozoic than during the rest of the Phanerozoic, consistent with extinction rates derived from paleontological databases. The high extinction susceptibility arises in the model from the limited geographical range of marine organisms. It stands even when assuming present-day pO2, suggesting that increasing oxygenation through the Paleozoic is not necessary to explain why extinction rates apparently declined with time.