Bioinspired In-Sensor Multimodal Fusion for Enhanced Spatial and Spatiotemporal Association

TLDR

Multimodal perception offers more precise and comprehensive information than unimodal approaches, yet existing systems merge signals only at computing terminals, risking loss of spatial association and requiring time stamps to preserve temporal coherence. The study demonstrates bioinspired in‑sensor multimodal fusion that enhances perception and reduces data transfer between sensors and computation units. The authors employ floating‑gate phototransistors with reconfigurable photoresponse plasticity to fuse visual–tactile and visual–audio signals in spatial and spatiotemporal modes, enabling real‑time dance‑music synchronization without time‑stamping. The approach simplifies multimodal integration and extends the in‑sensor computing paradigm.

Abstract

Multimodal perception can capture more precise and comprehensive information compared with unimodal approaches. However, current sensory systems typically merge multimodal signals at computing terminals following parallel processing and transmission, which results in the potential loss of spatial association information and requires time stamps to maintain temporal coherence for time-series data. Here we demonstrate bioinspired in-sensor multimodal fusion, which effectively enhances comprehensive perception and reduces the level of data transfer between sensory terminal and computation units. By adopting floating gate phototransistors with reconfigurable photoresponse plasticity, we realize the agile spatial and spatiotemporal fusion under nonvolatile and volatile photoresponse modes. To realize an optimal spatial estimation, we integrate spatial information from visual–tactile signals. For dynamic events, we capture and fuse in real time spatiotemporal information from visual–audio signals, realizing a dance-music synchronization recognition task without a time-stamping process. This in-sensor multimodal fusion approach provides the potential to simplify the multimodal integration system, extending the in-sensor computing paradigm.