Direct Human-AI Comparison in the Animal-AI Environment

Konstantinos Voudouris; Matthew Crosby; Benjamin Beyret; José Hernández-Orallo; Murray Shanahan; Marta Halina; Lucy G Cheke

doi:10.3389/fpsyg.2022.711821

Direct Human-AI Comparison in the Animal-AI Environment

Front Psychol. 2022 May 24:13:711821. doi: 10.3389/fpsyg.2022.711821. eCollection 2022.

Authors

Konstantinos Voudouris^{1

2}, Matthew Crosby^{1

3}, Benjamin Beyret^{1

3}, José Hernández-Orallo^{1

4}, Murray Shanahan^{1

3}, Marta Halina^{1

2

5}, Lucy G Cheke^{1

2}

Affiliations

¹ Leverhulme Centre for the Future of Intelligence, University of Cambridge, Cambridge, United Kingdom.
² Department of Psychology, University of Cambridge, Cambridge, United Kingdom, Cambridge, United Kingdom.
³ Department of Computing, Imperial College London, London, United Kingdom.
⁴ Valencian Research Institute for Artificial Intelligence (VRAIN), Universitat Politècnica de València, València, Spain.
⁵ Department of History and Philosophy of Science, University of Cambridge, Cambridge, United Kingdom.

Abstract

Artificial Intelligence is making rapid and remarkable progress in the development of more sophisticated and powerful systems. However, the acknowledgement of several problems with modern machine learning approaches has prompted a shift in AI benchmarking away from task-oriented testing (such as Chess and Go) towards ability-oriented testing, in which AI systems are tested on their capacity to solve certain kinds of novel problems. The Animal-AI Environment is one such benchmark which aims to apply the ability-oriented testing used in comparative psychology to AI systems. Here, we present the first direct human-AI comparison in the Animal-AI Environment, using children aged 6-10 (n = 52). We found that children of all ages were significantly better than a sample of 30 AIs across most of the tests we examined, as well as performing significantly better than the two top-scoring AIs, "ironbar" and "Trrrrr," from the Animal-AI Olympics Competition 2019. While children and AIs performed similarly on basic navigational tasks, AIs performed significantly worse in more complex cognitive tests, including detour tasks, spatial elimination tasks, and object permanence tasks, indicating that AIs lack several cognitive abilities that children aged 6-10 possess. Both children and AIs performed poorly on tool-use tasks, suggesting that these tests are challenging for both biological and non-biological machines.

Keywords: AI benchmarks; Animal-AI Olympics; artificial intelligence; cognitive AI; comparative cognition; human-AI comparison; out-of-distribution testing.