The spread of attention across modalities and space in a multisensory object

TLDR

Attending to a stimulus enhances its neural responses, and object‑based visual attention can automatically boost other parts of the same object, while audiovisual integration often mislocalizes sound toward vision. The study examined whether attending to a visual component of a multisensory object can modulate neural responses to its synchronous, spatially disparate auditory counterpart. ERP and fMRI showed that sounds synchronous with an attended visual stimulus produced larger, late, frontally distributed activity and increased auditory‑cortex activation, demonstrating that attention to one modality spreads to task‑irrelevant, spatially disparate signals in another modality as part of a multisensory object.

Abstract

Attending to a stimulus is known to enhance the neural responses to that stimulus. Recent experiments on visual attention have shown that this modulation can have object-based characteristics, such that, when certain parts of a visual object are attended, other parts automatically also receive enhanced processing. Here, we investigated whether visual attention can modulate neural responses to other components of a multisensory object defined by synchronous, but spatially disparate, auditory and visual stimuli. The audiovisual integration of such multisensory stimuli typically leads to mislocalization of the sound toward the visual stimulus (ventriloquism illusion). Using event-related potentials and functional MRI, we found that the brain's response to task-irrelevant sounds occurring synchronously with a visual stimulus from a different location was larger when that accompanying visual stimulus was attended versus unattended. The event-related potential effect consisted of sustained, frontally distributed, brain activity that emerged relatively late in processing, an effect resembling attention-related enhancements seen at earlier latencies during intramodal auditory attention. Moreover, the functional MRI data confirmed that the effect included specific enhancement of activity in auditory cortex. These findings indicate that attention to one sensory modality can spread to encompass simultaneous signals from another modality, even when they are task-irrelevant and from a different location. This cross-modal attentional spread appears to reflect an object-based, late selection process wherein spatially discrepant auditory stimulation is grouped with synchronous attended visual input into a multisensory object, resulting in the auditory information being pulled into the attentional spotlight and bestowed with enhanced processing.