We previously showed that impressions of nine semantic words expressing abstract meanings (like "tranquil") can be expressed by 12 hues in a paired comparison method; in this study, White, Gray, and Black were added (Exp. 1) to the previous 12 hues. Color impressions were also estimated using a set of 35 paired words by a semantic differential (SD) method (Exp. 2). The data of nine color vision normal (CVN) and seven color vision deficient (CVD) observers (one protanope and six deuteranopes) were analyzed separately by principal component analysis (PCA). In the results of Exp. 1, all hues used as loadings were distributed in a hue-circle shape in the 2D color space of PC axes for both observer groups [however, the four bluish hues (Blue-Green to Violet) tended toward convergence]. One data set of five CVNs and five deuteranopes was analyzed together using PCA because of high concordance. In the word distribution of the CVDs in Exp. 1, because second PC scores tended to be smaller, the categorization of the words was not clear; the points of five word scores were approximately on one line, reflecting that the colors used in the paired comparison were treated in one-dimensional scaling (which correlates to lightness) in the CVDs. In the results of Exp. 2, the word distribution of loadings was similar between the CVNs and CVDs, and the color score distribution had a similar tendency of showing an ellipse-shaped hue circle; it was probably caused by their experience of being associated with color names rather than color appearance (although the radius of the short axis is shorter in the CVDs' data). The comparison of the word distribution between experiments suggests that two to five semantic word impressions can be stably expressed by hue, but the impression of other words, such as "Magnificent" for the CVNs and "Fine" for the CVDs, cannot. The hue circle is conceptually kept in the SD measurement for all observers; however, it was not kept in the paired comparison for the CVDs. The analysis of one combined data set suggests that the 2D color distribution is not caused by a 3D color system because the lightness scaling is involved in the 2D color distribution.