Effect of recurrent infomax on the information processing capability of input-driven recurrent neural networks

Takuma Tanaka; Kohei Nakajima; Toshio Aoyagi

doi:10.1016/j.neures.2020.02.001

Effect of recurrent infomax on the information processing capability of input-driven recurrent neural networks

Neurosci Res. 2020 Jul:156:225-233. doi: 10.1016/j.neures.2020.02.001. Epub 2020 Feb 14.

Authors

Takuma Tanaka¹, Kohei Nakajima², Toshio Aoyagi³

Affiliations

¹ Graduate School of Data Science, Shiga University, 1-1-1 Banba, Hikone, Shiga 522-8522, Japan. Electronic address: takuma-tanaka@biwako.shiga-u.ac.jp.
² Graduate School of Information Science and Technology, University of Tokyo, Tokyo 113-8656, Japan.
³ Graduate School of Informatics, Kyoto University, Yoshida Honmachi, Sakyo-ku, Kyoto 606-8501, Japan.

PMID: 32068068
DOI: 10.1016/j.neures.2020.02.001

Abstract

Reservoir computing is a framework for exploiting the inherent transient dynamics of recurrent neural networks (RNNs) as a computational resource. On the basis of this framework, much research has been conducted to evaluate the relationship between the dynamics of RNNs and the RNNs' information processing capability. In this study, we present a detailed analysis of the information processing capability of an RNN optimized by recurrent infomax (RI), an unsupervised learning method that maximizes the mutual information of RNNs by adjusting the connection weights of the network. The results indicate that RI leads to the emergence of a delay-line structure and that the network optimized by the RI possesses a superior short-term memory, which is the ability to store the temporal information of the input stream in its transient dynamics.

Keywords: Memory; Oscillology; Recurrent infomax; Reservoir computing; Unsupervised learning.

MeSH terms

Cognition
Memory, Short-Term*
Neural Networks, Computer*