The point of attachment of spindle microtubules to metaphase chromosomes is known as the centromere. Plant and animal centromeres are epigenetically specified by a centromere-specific variant of Histone H3, CENH3 (a.k.a. CENP-A). Unlike canonical histones that are invariant, CENH3 proteins are accumulating substitutions at an accelerated rate. This diversification of CENH3 is a conundrum since its role as the key determinant of centromere identity remains a constant across species. Here, we ask whether naturally occurring divergence in CENH3 has functional consequences. We performed functional complementation assays on cenh3-1, a null mutation in Arabidopsis thaliana, using untagged CENH3s from increasingly distant relatives. Contrary to previous results using GFP-tagged CENH3, we find that the essential functions of CENH3 are conserved across a broad evolutionary landscape. CENH3 from a species as distant as the monocot Zea mays can functionally replace A. thaliana CENH3. Plants expressing variant CENH3s that are fertile when selfed show dramatic segregation errors when crossed to a wild-type individual. The progeny of this cross include hybrid diploids, aneuploids with novel genetic rearrangements and haploids that inherit only the genome of the wild-type parent. Importantly, it is always chromosomes from the plant expressing the divergent CENH3 that missegregate. Using chimeras, we show that it is divergence in the fast-evolving N-terminal tail of CENH3 that is causing segregation errors and genome elimination. Furthermore, we analyzed N-terminal tail sequences from plant CENH3s and discovered a modular pattern of sequence conservation. From this we hypothesize that while the essential functions of CENH3 are largely conserved, the N-terminal tail is evolving to adapt to lineage-specific centromeric constraints. Our results demonstrate that this lineage-specific evolution of CENH3 causes inviability and sterility of progeny in crosses, at the same time producing karyotypic variation. Thus, CENH3 evolution can contribute to postzygotic reproductive barriers.