Grading by AI makes me feel fairer? How different evaluators affect college students' perception of fairness

Abstract

Introduction: In the field of education, new technologies have enhanced the objectivity and scientificity of educational evaluation. However, concerns have been raised about the fairness of evaluators, such as artificial intelligence (AI) algorithms. This study aimed to assess college students' perceptions of fairness in educational evaluation scenarios through three studies using experimental vignettes.

Methods: Three studies were conducted involving 172 participants in Study 1, 149 in Study 2, and 145 in Study 3. Different evaluation contexts were used in each study to assess the influence of evaluators on students' perception of fairness. Information transparency and explanations for evaluation outcomes were also examined as potential moderators.

Results: Study 1 found that different evaluators could significantly influence the perception of fairness under three evaluation contexts. Students perceived AI algorithms as fairer evaluators than teachers. Study 2 revealed that information transparency was a mediator, indicating that students perceived higher fairness with AI algorithms due to increased transparency compared with teachers. Study 3 revealed that the explanation of evaluation outcomes moderated the effect of evaluator on students' perception of fairness. Specifically, when provided with explanations for evaluation results, the effect of evaluator on students' perception of fairness was lessened.

Discussion: This study emphasizes the importance of information transparency and comprehensive explanations in the evaluation process, which is more crucial than solely focusing on the type of evaluators. It also draws attention to potential risks like algorithmic hegemony and advocates for ethical considerations, including privacy regulations, in integrating new technologies into educational evaluation systems. Overall, this study provides valuable theoretical insights and practical guidance for conducting fairer educational evaluations in the era of new technologies.

Keywords: AI algorithm; explanation; fairness perception; higher education evaluation; information transparency.