Resolving multisensory conflict: a strategy for balancing the costs and benefits of audio-visual integration

TLDR

The brain continuously integrates sensory signals to form a coherent percept, typically modeled as a weighted average where each modality contributes proportionally to its reliability. The study examines how listeners judge auditory or visual rates while ignoring conflicting rates in the other modality, and introduces a Bayesian model to explain the observed behavior. The authors model integration as a Bayesian process that incorporates prior knowledge of auditory–visual rate correspondence to determine how much to integrate. Results reveal a gradual shift from partial integration to full segregation as inter‑modal discrepancy grows, contradicting strict maximum‑likelihood estimation, and suggest a strategy that balances integration benefits against the costs of mixing independent signals.

Abstract

In order to maintain a coherent, unified percept of the external environment, the brain must continuously combine information encoded by our different sensory systems. Contemporary models suggest that multisensory integration produces a weighted average of sensory estimates, where the contribution of each system to the ultimate multisensory percept is governed by the relative reliability of the information it provides (maximum-likelihood estimation). In the present study, we investigate interactions between auditory and visual rate perception, where observers are required to make judgments in one modality while ignoring conflicting rate information presented in the other. We show a gradual transition between partial cue integration and complete cue segregation with increasing inter-modal discrepancy that is inconsistent with mandatory implementation of maximum-likelihood estimation. To explain these findings, we implement a simple Bayesian model of integration that is also able to predict observer performance with novel stimuli. The model assumes that the brain takes into account prior knowledge about the correspondence between auditory and visual rate signals, when determining the degree of integration to implement. This provides a strategy for balancing the benefits accrued by integrating sensory estimates arising from a common source, against the costs of conflating information relating to independent objects or events.