Hybrid Macro/Micro Level Backpropagation for Training Deep Spiking\n Neural Networks

Yingyezhe Jin, Wenrui Zhang, Peng Li

arXiv (Cornell University) · 2018 · 116 citations · 0 references

DOIFull text

Open access

Concepts

Abstract

Spiking neural networks (SNNs) are positioned to enable spatio-temporal\ninformation processing and ultra-low power event-driven neuromorphic hardware.\nHowever, SNNs are yet to reach the same performances of conventional deep\nartificial neural networks (ANNs), a long-standing challenge due to complex\ndynamics and non-differentiable spike events encountered in training. The\nexisting SNN error backpropagation (BP) methods are limited in terms of\nscalability, lack of proper handling of spiking discontinuities, and/or\nmismatch between the rate-coded loss function and computed gradient. We present\na hybrid macro/micro level backpropagation (HM2-BP) algorithm for training\nmulti-layer SNNs. The temporal effects are precisely captured by the proposed\nspike-train level post-synaptic potential (S-PSP) at the microscopic level. The\nrate-coded errors are defined at the macroscopic level, computed and\nback-propagated across both macroscopic and microscopic levels. Different from\nexisting BP methods, HM2-BP directly computes the gradient of the rate-coded\nloss function w.r.t tunable parameters. We evaluate the proposed HM2-BP\nalgorithm by training deep fully connected and convolutional SNNs based on the\nstatic MNIST [14] and dynamic neuromorphic N-MNIST [26]. HM2-BP achieves an\naccuracy level of 99.49% and 98.88% for MNIST and N-MNIST, respectively,\noutperforming the best reported performances obtained from the existing SNN BP\nalgorithms. Furthermore, the HM2-BP produces the highest accuracies based on\nSNNs for the EMNIST [3] dataset, and leads to high recognition accuracy for the\n16-speaker spoken English letters of TI46 Corpus [16], a challenging\npatio-temporal speech recognition benchmark for which no prior success based on\nSNNs was reported. It also achieves competitive performances surpassing those\nof conventional deep learning models when dealing with asynchronous spiking\nstreams.\n