Bayesian Reinforcement Learning With Limited Cognitive Load

Dilip Arumugam; Mark K Ho; Noah D Goodman; Benjamin Van Roy

doi:10.1162/opmi_a_00132

Bayesian Reinforcement Learning With Limited Cognitive Load

Open Mind (Camb). 2024 Apr 3:8:395-438. doi: 10.1162/opmi_a_00132. eCollection 2024.

Authors

Dilip Arumugam¹, Mark K Ho², Noah D Goodman^{1

3}, Benjamin Van Roy^{4

5}

Affiliations

¹ Department of Computer Science, Stanford University.
² Center for Data Science, New York University.
³ Department of Psychology, Stanford University.
⁴ Department of Electrical Engineering, Stanford University.
⁵ Department of Management Science & Engineering, Stanford University.

Abstract

All biological and artificial agents must act given limits on their ability to acquire and process information. As such, a general theory of adaptive behavior should be able to account for the complex interactions between an agent's learning history, decisions, and capacity constraints. Recent work in computer science has begun to clarify the principles that shape these dynamics by bridging ideas from reinforcement learning, Bayesian decision-making, and rate-distortion theory. This body of work provides an account of capacity-limited Bayesian reinforcement learning, a unifying normative framework for modeling the effect of processing constraints on learning and action selection. Here, we provide an accessible review of recent algorithms and theoretical results in this setting, paying special attention to how these ideas can be applied to studying questions in the cognitive and behavioral sciences.

Keywords: Bayesian decision making; efficient exploration; information theory; multi-armed bandits; rate-distortion theory; reinforcement learning.