Generative language reconstruction from brain recordings

Abstract

Language reconstruction from non-invasive brain recordings has been a long-standing challenge. Existing research has addressed this challenge with a classification setup, where a set of language candidates are pre-constructed and then matched with the representation decoded from brain recordings. Here, we propose a method that addresses language reconstruction through auto-regressive generation, which directly uses the representation decoded from functional magnetic resonance imaging (fMRI) as the input for a large language model (LLM), mitigating the need for pre-constructed candidates. While an LLM can already generate high-quality content, our approach produces results more closely aligned with the visual or auditory language stimuli in response to which brain recordings are sampled, especially for content deemed "surprising" for the LLM. Furthermore, we show that the proposed approach can be used in an auto-regressive manner to reconstruct a 10 min-long language stimulus. Our method outperforms or is comparable to previous classification-based methods under different task settings, with the added benefit of estimating the likelihood of generating any semantic content. Our findings demonstrate the effectiveness of employing brain language interfaces in a generative setup and delineate a powerful and efficient means for mapping functional representations of language perception in the brain.