Wide crosses result in postzygotic elimination of one parental chromosome set, but the mechanisms that result in such differential fate are poorly understood. Here, we show that alterations of centromeric histone H3 (CENH3) lead to its selective removal from centromeres of mature Arabidopsis eggs and early zygotes, while wild-type CENH3 persists. In the hybrid zygotes and embryos, CENH3 and essential centromere proteins load preferentially on the CENH3-rich centromeres of the wild-type parent, while CENH3-depleted centromeres fail to reconstitute new CENH3-chromatin and the kinetochore and are frequently lost. Genome elimination is opposed by E3 ubiquitin ligase VIM1. We propose a model based on cooperative binding of CENH3 to chromatin to explain the differential CENH3 loading rates. Thus, parental CENH3 polymorphisms result in epigenetically distinct centromeres that instantiate a strong mating barrier and produce haploids.