In-sensor image memorization and encoding via optical neurons for bio-stimulus domain reduction toward visual cognitive processing

TLDR

Machine‑vision sensors generate large data volumes, and in‑sensor computing has emerged to reduce data transfer and enable fast, energy‑efficient visual cognition, yet current systems cannot process stored images directly within the sensor. We aim to demonstrate a heterogeneously integrated 1‑photodiode/1‑memristor (1P‑1R) crossbar that emulates mammalian image‑encoding to extract features from input images. The crossbar applies trained weight values as input voltage to the image‑saved array rather than storing them in memristors, enabling in‑sensor computing. The platform offers an advanced architecture for real‑time, data‑intensive machine‑vision applications by reducing the bio‑stimulus domain.

Abstract

As machine vision technology generates large amounts of data from sensors, it requires efficient computational systems for visual cognitive processing. Recently, in-sensor computing systems have emerged as a potential solution for reducing unnecessary data transfer and realizing fast and energy-efficient visual cognitive processing. However, they still lack the capability to process stored images directly within the sensor. Here, we demonstrate a heterogeneously integrated 1-photodiode and 1 memristor (1P-1R) crossbar for in-sensor visual cognitive processing, emulating a mammalian image encoding process to extract features from the input images. Unlike other neuromorphic vision processes, the trained weight values are applied as an input voltage to the image-saved crossbar array instead of storing the weight value in the memristors, realizing the in-sensor computing paradigm. We believe the heterogeneously integrated in-sensor computing platform provides an advanced architecture for real-time and data-intensive machine-vision applications via bio-stimulus domain reduction.