Categorical color constancy in normal trichromats has been found to be very robust in real scenes. In this study, we investigated categorical color constancy in red-green dichromats and anomalous trichromats. Eight dichromats (two protanopes and six deuteranopes), eight anomalous trichromats (four protanomalous and four deuteranomalous trichromats), and eight normal trichromats sorted 208 Munsell matte surfaces into Berlin and Kay's basic color categories under D65 illuminant, F illuminant with correlated color temperature 4200 K, and TL84 illuminant with correlated color temperature 2700 K. Color constancy was quantified by a color constancy index. The results showed that the constancy index of dichromats (0.79) was considerable and significantly lower than that of normal trichromats (0.87) while that of anomalous trichromats (0.84) was not. The impairment of color constancy performance in dichromats was expected to be caused by their large intra-subject variabilities in color naming. The results indicate robust categorical color constancy along daylight locus in red-green dichromats and anomalous trichromats, which might be contributed by cone adaptation mechanism and be independent of color discrimination mechanism. It suggests that the color categorization by color vision deficient subjects can be reasonable without any assistants of artificial equipment in daily life under sunlight and common illuminations.