Nano Letters · 2024 · 28 citations · 38 references
ComputingNeuroscienceSensor FusionSensorsVisual PerceptionMultimodal InteractionMultisensor Data IntegrationMultisensory IntegrationSensor IntegrationSensory IntegrationTime SynchronizationMultimodal Data FusionMultimodal FusionPhototransistorsSpatial EstimationSpatiotemporal Data FusionTemporal CoherenceTime-stampingVisual Signals
Multimodal perception offers more precise and comprehensive information than unimodal approaches, yet existing systems merge signals only at computing terminals, risking loss of spatial association and requiring time stamps to preserve temporal coherence. The study demonstrates bioinspired in‑sensor multimodal fusion that enhances perception and reduces data transfer between sensors and computation units. The authors employ floating‑gate phototransistors with reconfigurable photoresponse plasticity to fuse visual–tactile and visual–audio signals in spatial and spatiotemporal modes, enabling real‑time dance‑music synchronization without time‑stamping. The approach simplifies multimodal integration and extends the in‑sensor computing paradigm.
Multimodal perception can capture more precise and comprehensive information compared with unimodal approaches. However, current sensory systems typically merge multimodal signals at computing terminals following parallel processing and transmission, which results in the potential loss of spatial association information and requires time stamps to maintain temporal coherence for time-series data. Here we demonstrate bioinspired in-sensor multimodal fusion, which effectively enhances comprehensive perception and reduces the level of data transfer between sensory terminal and computation units. By adopting floating gate phototransistors with reconfigurable photoresponse plasticity, we realize the agile spatial and spatiotemporal fusion under nonvolatile and volatile photoresponse modes. To realize an optimal spatial estimation, we integrate spatial information from visual–tactile signals. For dynamic events, we capture and fuse in real time spatiotemporal information from visual–audio signals, realizing a dance-music synchronization recognition task without a time-stamping process. This in-sensor multimodal fusion approach provides the potential to simplify the multimodal integration system, extending the in-sensor computing paradigm.
38
A bioinspired flexible organic artificial afferent nerve
Yeongin Kim, Alex Chortos, Wentao Xu et al. · Science · 2018 · 1.4K citations
Optoelectronic resistive random access memory for neuromorphic vision sensors
Feichi Zhou, Zheng Zhou, Jiewei Chen et al. · Nature Nanotechnology · 2019 · 1.3K citations
Near-sensor and in-sensor computing
Feichi Zhou, Yang Chai · Nature Electronics · 2020 · 1K citations · Full text
Neuronal Oscillations and Multisensory Interaction in Primary Auditory Cortex
Péter Lakatos, Chi-Ming Chen, Monica N. O’Connell et al. · Neuron · 2007 · 1K citations · Full text