A Heterogeneously Integrated Spiking Neuron Array for Multimode‐Fused Perception and Object Classification

TLDR

Multimode‑fused sensing in the somatosensory system enables comprehensive object property acquisition and accurate judgments, yet conventional CMOS technology faces significant integration and circuit complexity challenges. This work reports a compact multimode‑fused spiking neuron (MFSN) designed to emulate human‑like multisensory perception. The MFSN heterogeneously integrates a pressure sensor and a NbOx‑based memristor, and is fabricated into 3 × 3 and larger arrays whose fused frequency patterns are routed to a spiking neural network for tactile pattern recognition and object classification. The MFSN fuses analog pressure and temperature signals into a single spike train with excellent compression, distinguishes modalities via frequency and amplitude decoupling, and achieves accurate tactile pattern recognition and multi‑attribute object classification, demonstrating its potential for intelligent robotics.

Abstract

Multimode-fused sensing in the somatosensory system helps people obtain comprehensive object properties and make accurate judgments. However, building such multisensory systems with conventional metal-oxide-semiconductor technology presents serious device integration and circuit complexity challenges. Here, a multimode-fused spiking neuron (MFSN) with a compact structure to achieve human-like multisensory perception is reported. The MFSN heterogeneously integrates a pressure sensor to process pressure and a NbOx -based memristor to sense temperature. Using this MFSN, multisensory analog information can be fused into one spike train, showing excellent data compression and conversion capabilities. Moreover, both pressure and temperature information are distinguished from fused spikes by decoupling the output frequencies and amplitudes, supporting multimodal tactile perception. Then, a 3 × 3 MFSN array is fabricated, and the fused frequency patterns are fed into a spiking neural network for enhanced tactile pattern recognition. Finally, a larger MFSN array is simulated for classifying objects with different shapes, temperatures, and weights, validating the feasibility of the MFSNs for practical applications. The proof-of-concept MFSNs enable the building of multimodal sensory systems and contribute to the development of highly intelligent robotics.