Efficient Decoding of Compositional Structure in Holistic Representations

Denis Kleyko; Connor Bybee; Ping-Chen Huang; Christopher J Kymn; Bruno A Olshausen; E Paxon Frady; Friedrich T Sommer

doi:10.1162/neco_a_01590

Efficient Decoding of Compositional Structure in Holistic Representations

Neural Comput. 2023 Jun 12;35(7):1159-1186. doi: 10.1162/neco_a_01590.

Authors

Denis Kleyko^{1

2}, Connor Bybee³, Ping-Chen Huang⁴, Christopher J Kymn⁵, Bruno A Olshausen⁶, E Paxon Frady⁷, Friedrich T Sommer^{1

8}

Affiliations

¹ Redwood Center for Theoretical Neuroscience, University of California at Berkeley, Berkeley, CA 94720, U.S.A.
² Intelligent Systems Laboratory, Research Institutes of Sweden, 16440 Kista, Sweden denis.kleyko@ri.se.
³ Redwood Center for Theoretical Neuroscience, University of California at Berkeley, Berkeley, CA 94720, U.S.A. bybee@berkeley.edu.
⁴ Redwood Center for Theoretical Neuroscience, University of California at Berkeley, Berkeley, CA 94720, U.S.A. pingchen.huang@berkeley.edu.
⁵ Redwood Center for Theoretical Neuroscience, University of California at Berkeley, Berkeley, CA 94720, U.S.A. cjkymn@berkeley.edu.
⁶ Redwood Center for Theoretical Neuroscience, University of California at Berkeley, Berkeley, CA 94720, U.S.A. baolshausen@berkeley.edu.
⁷ Neuromorphic Computing Laboratory, Intel Labs, Santa Clara, CA 95054, U.S.A. e.paxon.frady@intel.com.
⁸ Neuromorphic Computing Laboratory, Intel Labs, Santa Clara, CA 95054, U.S.A. fsommer@berkeley.edu.

PMID: 37187162
DOI: 10.1162/neco_a_01590

Abstract

We investigate the task of retrieving information from compositional distributed representations formed by hyperdimensional computing/vector symbolic architectures and present novel techniques that achieve new information rate bounds. First, we provide an overview of the decoding techniques that can be used to approach the retrieval task. The techniques are categorized into four groups. We then evaluate the considered techniques in several settings that involve, for example, inclusion of external noise and storage elements with reduced precision. In particular, we find that the decoding techniques from the sparse coding and compressed sensing literature (rarely used for hyperdimensional computing/vector symbolic architectures) are also well suited for decoding information from the compositional distributed representations. Combining these decoding techniques with interference cancellation ideas from communications improves previously reported bounds (Hersche et al., 2021) of the information rate of the distributed representations from 1.20 to 1.40 bits per dimension for smaller codebooks and from 0.60 to 1.26 bits per dimension for larger codebooks.

Grants and funding

R01 EB026955/EB/NIBIB NIH HHS/United States