A long-standing paradox of marine populations is chaotic genetic patchiness (CGP), temporally unstable patterns of genetic differentiation that occur below the geographic scale of effective dispersal. Several mechanisms are hypothesized to explain CGP including natural selection, spatiotemporal fluctuations in larval source populations, self-recruitment, and sweepstake reproduction. Discriminating among them is extremely difficult but is fundamental to understanding how marine organisms reproduce and disperse. Here, we report a notable example of CGP in the Antarctic limpet, an unusually tractable system where multiple confounding explanations can be discounted. Using population genomics, temporally replicated sampling, surface drifters, and forward genetic simulations, we show that CGP likely arises from an extreme sweepstake event together with collective larval dispersal, while selection appears to be unimportant. Our results illustrate the importance of neutral demographic forces in natural populations and have important implications for understanding the recruitment dynamics, population connectivity, local adaptation, and resilience of marine populations.