Multimodal In‐Sensor Computing Implemented by Easily‐Fabricated Oxide‐Heterojunction Optoelectronic Synapses

TLDR

The advent of big data and the Internet of Things has created urgent demands for in‑sensor computing hardware with multimodal perception that can resolve inefficiency, high latency, and excessive energy consumption in conventional sensory systems. This work proposes a simple‑structured optoelectronic synaptic device based on an In₂O₃·SnO₂/Nb:SrTiO₃ heterostructure that demonstrates in‑sensor computing and multimodal perception capabilities. The device’s optical and electrical synaptic responses enable a multimodal neuromorphic system that simultaneously perceives visual and auditory signals, and its integrated sensing, processing, and dynamic memory functions allow a neuromorphic vision system to monitor moving vehicles in real time with high accuracy. The multimodal system successfully avoids misjudgments in human emotion recognition that arise from single‑modal cognition, and the low‑cost, easily mass‑produced optoelectronic synaptic device paves the way for next‑generation in‑sensor computing capable of efficiently processing multimodal signals.

Abstract

Abstract The advent of big data and the Internet of Things has created urgent demands for in‐sensor computing hardware with multimodal perception that can effectively resolve the inefficiency, high latency, and excessive energy consumption challenges faced by conventional sensory systems. Here, a simple‐structured optoelectronic synaptic device of In 2 O 3 ·SnO 2 /Nb:SrTiO 3 (ITO/NSTO) heterostructure is proposed, which vividly demonstrates in‐sensor computing and multimodal perception capabilities. First, with the ingenious synaptic responses of the device under both optical and electrical stimuli, a multimodal in‐sensor neuromorphic computing system capable of concurrently perceiving and processing visual and auditory information is constructed. Using this multimodal system to perform a human emotion recognition task, the misjudgment arising from single‐modal cognition can thereby be effectively avoided. Second, utilizing the integrated sensing and processing functions, along with the dynamic memories of the device array, a neuromorphic vision system is implemented for real‐time monitoring of moving vehicles, which displays high recognition efficiency and accuracy. This research not only provides an optoelectronic synaptic device that is low‐cost and easy to mass‐produce, but also paves the way for next‐generation in‐sensor computing that can efficiently process multimodal signals.