Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI

Richard Tomsett; Alun Preece; Dave Braines; Federico Cerutti; Supriyo Chakraborty; Mani Srivastava; Gavin Pearson; Lance Kaplan

doi:10.1016/j.patter.2020.100049

Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI

Patterns (N Y). 2020 Jul 10;1(4):100049. doi: 10.1016/j.patter.2020.100049.

Authors

Richard Tomsett¹, Alun Preece², Dave Braines¹, Federico Cerutti^{2

3}, Supriyo Chakraborty⁴, Mani Srivastava⁵, Gavin Pearson⁶, Lance Kaplan⁷

Affiliations

¹ Emerging Technology, IBM Research Europe, Hursley Park Road, Hursley SO21 2JN, UK.
² Crime and Security Research Institute, Cardiff University, Friary House, Greyfriars Road, Cardiff CF10 3AE, UK.
³ Dipartimento di Ingegneria dell'Informazione, Università degli Studi di Brescia, Via Branze 38, Brescia 25123, Italy.
⁴ IBM Research, IBM Thomas J. Watson Research Center, 1101 Kitchawan Road, Yorktown Heights, NY 10598, USA.
⁵ Networked and Embedded Systems Laboratory, Electrical and Computer Engineering Department, University of California, Los Angeles, 420 Westwood Plaza, Los Angeles, CA 90095-1594, USA.
⁶ Defence Science and Technology Laboratory, Porton Down, Salisbury, Wiltshire SP4 0JQ, UK.
⁷ CCDC Army Research Laboratory, 2800 Powder Mill Road, Adelphi, MD 20783, USA.

Abstract

Artificial intelligence (AI) systems hold great promise as decision-support tools, but we must be able to identify and understand their inevitable mistakes if they are to fulfill this potential. This is particularly true in domains where the decisions are high-stakes, such as law, medicine, and the military. In this Perspective, we describe the particular challenges for AI decision support posed in military coalition operations. These include having to deal with limited, low-quality data, which inevitably compromises AI performance. We suggest that these problems can be mitigated by taking steps that allow rapid trust calibration so that decision makers understand the AI system's limitations and likely failures and can calibrate their trust in its outputs appropriately. We propose that AI services can achieve this by being both interpretable and uncertainty-aware. Creating such AI systems poses various technical and human factors challenges. We review these challenges and recommend directions for future research.

Keywords: DSML 1: Concept: Basic principles of a new data science output observed and reported.

Publication types

Review