ExpMRC: explainability evaluation for machine reading comprehension

Yiming Cui; Ting Liu; Wanxiang Che; Zhigang Chen; Shijin Wang

doi:10.1016/j.heliyon.2022.e09290

ExpMRC: explainability evaluation for machine reading comprehension

Heliyon. 2022 Apr 19;8(4):e09290. doi: 10.1016/j.heliyon.2022.e09290. eCollection 2022 Apr.

Authors

Yiming Cui^{1

2}, Ting Liu¹, Wanxiang Che¹, Zhigang Chen², Shijin Wang^{2

3}

Affiliations

¹ Research Center for SCIR, Harbin Institute of Technology, Harbin 150001, China.
² State Key Laboratory of Cognitive Intelligence, iFLYTEK Research, Beijing 100010, China.
³ iFLYTEK AI Research (Central China), Wuhan 430000, China.

Abstract

Achieving human-level performance on some Machine Reading Comprehension (MRC) datasets is no longer challenging with the help of powerful Pre-trained Language Models (PLMs). However, it is necessary to provide both answer prediction and its explanation to further improve the MRC system's reliability, especially for real-life applications. In this paper, we propose a new benchmark called ExpMRC for evaluating the textual explainability of the MRC systems. ExpMRC contains four subsets, including SQuAD, CMRC 2018, RACE⁺, and C³, with additional annotations of the answer's evidence. The MRC systems are required to give not only the correct answer but also its explanation. We use state-of-the-art PLMs to build baseline systems and adopt various unsupervised approaches to extract both answer and evidence spans without human-annotated evidence spans. The experimental results show that these models are still far from human performance, suggesting that the ExpMRC is challenging. Resources (data and baselines) are available through https://github.com/ymcui/expmrc.

Keywords: Explainable artificial intelligence; Machine reading comprehension; Natural language processing.