Color matching functions (CMFs), which are used to derive tristimulus values and chromaticities, play a very important role in color characterization, calibration, and specifications. Great efforts have been made to investigate how CMFs can characterize the color matches and mismatches between stimuli with different spectral compositions under different sizes of field of view (FOV). Few study specifically investigated how to better characterize the CMFs for different observer ages. In this study, we carefully designed a series of color stimuli using different CMFs based on our two past studies, and asked 51 young, middle-aged, and senior observers to evaluate the color differences. The three sets of CIE 2006 CMFs (i.e., 10°, 4°, and 2°) with an age of 20-, 40-, and 70-year were found to characterize the perceived color differences for the young, middle-aged, and senior observers. In addition, it was found that the characterization of a group of observers can also be performed using these three sets of CMFs based on the distributions of the observers, which had a better correlation to the perceived color differences than the individual colorimetric model.