Retrotransposons have invaded eukaryotic centromeres in cycles of repeat expansion and purging, but the function of centromeric retrotransposons, if any, has remained unclear. In Arabidopsis, centromeric ATHILA retrotransposons give rise to epigenetically activated short interfering RNAs (easiRNAs) in mutants in DECREASE IN DNA METHYLATION1 (DDM1), which promote histone H3 lysine-9 di-methylation (H3K9me2). Here, we show that mutants which lose both DDM1 and RNA dependent RNA polymerase (RdRP) have pleiotropic developmental defects and mis-segregation of chromosome 5 during mitosis. Fertility defects are epigenetically inherited with the centromeric region of chromosome 5, and can be rescued by directing artificial small RNAs to a single family of ATHILA5 retrotransposons specifically embedded within this centromeric region. easiRNAs and H3K9me2 promote pericentromeric condensation, chromosome cohesion and proper chromosome segregation in mitosis. Insertion of ATHILA silences transcription, while simultaneously making centromere function dependent on retrotransposon small RNAs, promoting the selfish survival and spread of centromeric retrotransposons. Parallels are made with the fission yeast S. pombe, where chromosome segregation depends on RNAi, and with humans, where chromosome segregation depends on both RNAi and HELLSDDM1.