Adaptive radiations often follow the evolution of key traits, such as the origin of the amniotic egg and the subsequent radiation of terrestrial vertebrates. The mechanism by which a species determines the sex of its offspring has been linked to critical ecological and life-history traits but not to major adaptive radiations, in part because sex-determining mechanisms do not fossilize. Here we establish a previously unknown coevolutionary relationship in 94 amniote species between sex-determining mechanism and whether a species bears live young or lays eggs. We use that relationship to predict the sex-determining mechanism in three independent lineages of extinct Mesozoic marine reptiles (mosasaurs, sauropterygians and ichthyosaurs), each of which is known from fossils to have evolved live birth. Our results indicate that each lineage evolved genotypic sex determination before acquiring live birth. This enabled their pelagic radiations, where the relatively stable temperatures of the open ocean constrain temperature-dependent sex determination in amniote species. Freed from the need to move and nest on land, extreme physical adaptations to a pelagic lifestyle evolved in each group, such as the fluked tails, dorsal fins and wing-shaped limbs of ichthyosaurs. With the inclusion of ichthyosaurs, mosasaurs and sauropterygians, genotypic sex determination is present in all known fully pelagic amniote groups (sea snakes, sirenians and cetaceans), suggesting that this mode of sex determination and the subsequent evolution of live birth are key traits required for marine adaptive radiations in amniote lineages.