Unveiling functions of the visual cortex using task-specific deep neural networks

PLoS Comput Biol. 2021 Aug 13;17(8):e1009267. doi: 10.1371/journal.pcbi.1009267. eCollection 2021 Aug.

Abstract

The human visual cortex enables visual perception through a cascade of hierarchical computations in cortical regions with distinct functionalities. Here, we introduce an AI-driven approach to discover the functional mapping of the visual cortex. We related human brain responses to scene images measured with functional MRI (fMRI) systematically to a diverse set of deep neural networks (DNNs) optimized to perform different scene perception tasks. We found a structured mapping between DNN tasks and brain regions along the ventral and dorsal visual streams. Low-level visual tasks mapped onto early brain regions, 3-dimensional scene perception tasks mapped onto the dorsal stream, and semantic tasks mapped onto the ventral stream. This mapping was of high fidelity, with more than 60% of the explainable variance in nine key regions being explained. Together, our results provide a novel functional mapping of the human visual cortex and demonstrate the power of the computational approach.

Publication types

  • Research Support, Non-U.S. Gov't

MeSH terms

  • Adult
  • Brain Mapping / statistics & numerical data*
  • Computational Biology
  • Deep Learning
  • Female
  • Functional Neuroimaging
  • Humans
  • Imaging, Three-Dimensional
  • Magnetic Resonance Imaging
  • Male
  • Models, Neurological
  • Neural Networks, Computer*
  • Photic Stimulation
  • Semantics
  • Task Performance and Analysis
  • Visual Cortex / anatomy & histology
  • Visual Cortex / diagnostic imaging
  • Visual Cortex / physiology*
  • Visual Perception / physiology

Grants and funding

G.R. thanks the support of the Alfons and Gertrud Kassel Foundation. R.M.C. is supported by DFG grants (CI241/1-1, CI241/3-1) and the ERC Starting Grant (ERC-2018- StG 803370). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.