Real-time classification and sensor fusion with a spiking deep belief network

TLDR

Deep Belief Networks excel at classification and offer generative insights, yet their parallel, feedback‑heavy structure makes them costly on serial computers. This work maps an offline‑trained DBN onto an efficient event‑driven spiking neural network using the Siegert approximation for Integrate‑and‑Fire neurons, enabling hardware implementation. The method is validated in simulation and on a laptop with a 3‑layer, 2694‑neuron network that fuses input from a 128 × 128 Dynamic Vision Sensor retina and a 64‑channel AER‑EAR cochlea, implemented in the open‑source jAER framework. The system achieves real‑time MNIST digit recognition with less than 1 % performance loss, an average latency of 5.8 ms, and improved disambiguation through cue integration from both sensors.

Abstract

Deep Belief Networks (DBNs) have recently shown impressive performance on a broad range of classification problems. Their generative properties allow better understanding of the performance, and provide a simpler solution for sensor fusion tasks. However, because of their inherent need for feedback and parallel update of large numbers of units, DBNs are expensive to implement on serial computers. This paper proposes a method based on the Siegert approximation for Integrate-and-Fire neurons to map an offline-trained DBN onto an efficient event-driven spiking neural network suitable for hardware implementation. The method is demonstrated in simulation and by a real-time implementation of a 3-layer network with 2694 neurons used for visual classification of MNIST handwritten digits with input from a 128 × 128 Dynamic Vision Sensor (DVS) silicon retina, and sensory-fusion using additional input from a 64-channel AER-EAR silicon cochlea. The system is implemented through the open-source software in the jAER project and runs in real-time on a laptop computer. It is demonstrated that the system can recognize digits in the presence of distractions, noise, scaling, translation and rotation, and that the degradation of recognition performance by using an event-based approach is less than 1%. Recognition is achieved in an average of 5.8 ms after the onset of the presentation of a digit. By cue integration from both silicon retina and cochlea outputs we show that the system can be biased to select the correct digit from otherwise ambiguous input.