Over the last ten years, satellite and geographically constrained in situ observations largely focused on the northern hemisphere have suggested that annual phytoplankton biomass cycles cannot be fully understood from environmental properties controlling phytoplankton division rates (e.g., nutrients and light), as they omit the role of ecological and environmental loss processes (e.g., grazing, viruses, sinking). Here, we use multi-year observations from a very large array of robotic drifting floats in the Southern Ocean to determine key factors governing phytoplankton biomass dynamics over the annual cycle. Our analysis reveals seasonal phytoplankton accumulation ('blooming') events occurring during periods of declining modeled division rates, an observation that highlights the importance of loss processes in dictating the evolution of the seasonal cycle in biomass. In the open Southern Ocean, the spring bloom magnitude is found to be greatest in areas with high dissolved iron concentrations, consistent with iron being a well-established primary limiting nutrient in this region. Under ice observations show that biomass starts increasing in early winter, well before sea ice begins to retreat. The average theoretical sensitivity of the Southern Ocean to potential changes in seasonal nutrient and light availability suggests that a 10% change in phytoplankton division rate may be associated with a 50% reduction in mean bloom magnitude and annual primary productivity, assuming simple changes in the seasonal magnitude of phytoplankton division rates. Overall, our results highlight the importance of quantifying and accounting for both division and loss processes when modeling future changes in phytoplankton biomass cycles.