Human decision making involving many alternatives is encumbered with inconsistent prioritization. Although inconsistency is assumed to grow with the number of comparisons, it is shown to be reduced by conscious awareness under certain conditions. This study experimentally investigated the effect of repeating a criteria ranking task on inconsistency scores as measured by four different inconsistency coefficients. A total of 107 participants were engaged in a selection task that comprised of ranking from 3 to 10 criteria and was repeated in three trials. Upon completing the first trial, the participants were informed about the inconsistency issues and could improve their ranking in another two trials. The inconsistency score was computed for each set of comparisons and the effect of repeating the selection task on inconsistency concerning the number of criteria was analyzed using the repeated measures ANOVA. The results reveal a significant change in the inconsistency as the task was repeated but the difference depended on the number of criteria. There exists a borderline in the problem size under which the rankings are associated with significantly lower inconsistency, while the rankings with the larger number of criteria were found to have significantly higher inconsistency.