We have developed a new method for selecting the test color sample set (TCSS) used to calculate CIE 2017 color fidelity index (CIE-Rf). Taking a Large Set as a starting point, a new optimized color sample set (OCSS) is obtained by clustering analysis. Taking metamerism phenomenon into account, spectra clustering is performed within the class obtained from color appearance attributes clustering. The CIE-Rf of 1202 light sources are calculated and analyzed by taking the Large Set, OCSS and CIE color evaluation sample set (CIE CESS-99) as TCSS. Through analyzing CIE-Rf, the performance of the OCSS is further investigated. The results show that the clustering analysis method developed in this paper can be well used in selecting test color samples, and the obtained OCSS can represent Large Set well and be better used for color fidelity metrics of light sources.