Osmanthus fragrans is a well-known ornamental plant that has been domesticated in China for 2500 years. More than 160 cultivars have been found during this long period of domestication, and they have subsequently been divided into four cultivar groups, including the Yingui, Jingui, Dangui, and Sijigui groups. These groups provide a set of materials to study genetic evolution and variability. Here, we constructed a reference genome of O. fragrans 'Liuyejingui' in the Jingui group and investigated its floral color traits and domestication history by resequencing a total of 122 samples, including 119 O. fragrans accessions and three other Osmanthus species, at an average sequencing depth of 15×. The population structure analysis showed that these 119 accessions formed an apparent regional cluster. The results of linkage disequilibrium (LD) decay analysis suggested that varieties with orange/red flower color in the Dangui group had undergone more artificial directional selection; these varieties had the highest LD values among the four groups, followed by the Sijigui, Jingui, and Yingui groups. Through a genome-wide association study, we further identified significant quantitative trait loci and genomic regions containing several genes, such as ethylene-responsive transcription factor 2 and Arabidopsis pseudoresponse regulator 2, that are positively associated with petal color. Moreover, we found a frameshift mutation with a 34-bp deletion in the first coding region of the carotenoid cleavage dioxygenase 4 gene. This frameshift mutation existed in at least one site on both alleles in all varieties of the Dangui group. The results from this study shed light on the genetic basis of domestication in woody plants, such as O. fragrans.