The pepper restorer-of-fertility (CaRf) gene was fine mapped based on conjoint analysis of recombinants and collinearity between the two pepper reference genomes. Capana06g003028, encoding an Rf-like PPR protein, was proposed as the strongest candidate for pepper CaRf based on sequence comparison and expression analysis. The cytoplasmic male sterility (CMS)/restorer-of-fertility (Rf) system not only provides an excellent model to dissect genetic interactions between the mitochondria and nucleus but also plays a vital role in high-efficiency hybrid seed production in crops including pepper (Capsicum spp.). Although two important CMS candidate genes, orf507 and Ψatp6-2, have previously been suggested, the pepper Rf gene (CaRf) has not yet been isolated. In this study, a high-density genetic map comprising 7566 SNP markers in 1944 bins was first constructed with the array genotyping results from 317 F2 individuals. Based on this map, the CaRf gene was preliminarily mapped to a region of 1.15 Mb in length at the end of chromosome P6. Then, by means of a conjoint analysis of recombinants and collinearity between the two pepper reference genomes, an important candidate interval with 270.10 kb in length was delimited for CaRf. Finally, Capana06g003028, which encodes an Rf-like PPR protein, was proposed as the strongest candidate for CaRf based on sequence analysis and expression characteristics in sterile and fertile plants. The high-density genetic map and fine mapping results provided here lay a foundation for the application of molecular breeding, as well as cloning and functional analysis of CaRf, in pepper.