Plant mating systems are known to influence population genetic structure because pollen and seed dispersal are often spatially restricted. However, the reciprocal outcomes of population structure on the dynamics of polymorphic mating systems have received little attention. In gynodioecious sea beet (Beta vulgaris ssp. maritima), three sexual types co-occur: females carrying a cytoplasmic male sterility (CMS) gene, hermaphrodites carrying a non-CMS cytoplasm and restored hermaphrodites that carry CMS genes and nuclear restorer alleles. This study investigated the effects of fine-scale genetic structure on male reproductive success of the two hermaphroditic forms. Our study population was strongly structured and characterized by contrasting local sex-ratios. Pollen flow was constrained over short distances and depended on local plant density. Interestingly, restored hermaphrodites sired significantly more seedlings than non-CMS hermaphrodites, despite the previous observation that the former produce pollen of lower quality than the latter. This result was explained by the higher frequency of females in the local vicinity of restored (CMS) hermaphrodites as compared to non-CMS hermaphrodites. Population structure thus strongly influences individual fitness and may locally counteract the expected effects of selection, suggesting that understanding fine scale population processes is central to predicting the evolution of gender polymorphism in angiosperms.