Bimodal speech: early suppressive visual effects in human auditory cortex

TLDR

Seeing lip movements can improve speech perception, but the neural mechanisms of audiovisual speech integration remain unclear. ERPs were recorded during an auditory recognition task with syllables presented in auditory, visual, or congruent bimodal conditions. Bimodal syllables were identified faster than auditory alone, associated with cross‑modal ERP effects (AV−(A+V)) around 120–190 ms, mainly reflecting decreased unimodal N1 activity in auditory cortex, indicating suppressive, speech‑specific audiovisual integration occurring at pre‑representational stages likely via feedback from visual or polymodal areas.

Abstract

Abstract While everyone has experienced that seeing lip movements may improve speech perception, little is known about the neural mechanisms by which audiovisual speech information is combined. Event‐related potentials (ERPs) were recorded while subjects performed an auditory recognition task among four different natural syllables randomly presented in the auditory (A), visual (V) or congruent bimodal (AV) condition. We found that: (i) bimodal syllables were identified more rapidly than auditory alone stimuli; (ii) this behavioural facilitation was associated with cross‐modal [AV − (A + V)] ERP effects around 120–190 ms latency, expressed mainly as a decrease of unimodal N1 generator activities in the auditory cortex. This finding provides evidence for suppressive, speech‐specific audiovisual integration mechanisms, which are likely to be related to the dominance of the auditory modality for speech perception. Furthermore, the latency of the effect indicates that integration operates at pre‐representational stages of stimulus analysis, probably via feedback projections from visual and/or polymodal areas.