Performance-optimized hierarchical models predict neural responses in higher visual cortex

TLDR

Humans and monkeys recognize objects effortlessly, a skill supported by a hierarchically interconnected brain network, yet deciphering neurons in higher visual hierarchy remains a major challenge. The study aims to identify a neural network model that matches human performance on challenging object categorization tasks. Using computational techniques, the authors develop a neural network model that replicates human performance on these tasks. The model, though not trained on neural data, accurately predicts responses in V4 and IT, and the findings imply that biological performance optimization directly shaped neural mechanisms.

Abstract

Significance Humans and monkeys easily recognize objects in scenes. This ability is known to be supported by a network of hierarchically interconnected brain areas. However, understanding neurons in higher levels of this hierarchy has long remained a major challenge in visual systems neuroscience. We use computational techniques to identify a neural network model that matches human performance on challenging object categorization tasks. Although not explicitly constrained to match neural data, this model turns out to be highly predictive of neural responses in both the V4 and inferior temporal cortex, the top two layers of the ventral visual hierarchy. In addition to yielding greatly improved models of visual cortex, these results suggest that a process of biological performance optimization directly shaped neural mechanisms.