Backpropagation for energy-efficient neuromorphic computing

TLDR

Embedded neural networks require high‑performance training algorithms and energy‑efficient hardware, with backpropagation delivering state‑of‑the‑art accuracy and neuromorphic chips offering unprecedented energy efficiency. The study aims to reconcile backpropagation’s continuous‑output neurons and weights with neuromorphic spiking neurons and discrete synapses. We treat spikes and discrete synapses as continuous probabilities, train with standard backpropagation, then map the trained network to neuromorphic hardware by sampling the probabilities to generate multiple networks that are merged through ensemble averaging, as demonstrated on a sparsely connected TrueNorth network trained on MNIST. An ensemble of 64 achieves 99.42 % accuracy at 108 µJ per image, while a single‑network ensemble achieves 92.7 % accuracy at 0.268 µJ per image.

Abstract

Solving real world problems with embedded neural networks requires both training algorithms that achieve high performance and compatible hardware that runs in real time while remaining energy efficient. For the former, deep learning using backpropagation has recently achieved a string of successes across many domains and datasets. For the latter, neuromorphic chips that run spiking neural networks have recently achieved unprecedented energy efficiency. To bring these two advances together, we must first resolve the incompatibility between backpropagation, which uses continuous-output neurons and synaptic weights, and neuromorphic designs, which employ spiking neurons and discrete synapses. Our approach is to treat spikes and discrete synapses as continuous probabilities, which allows training the network using standard backpropagation. The trained network naturally maps to neuromorphic hardware by sampling the probabilities to create one or more networks, which are merged using ensemble averaging. To demonstrate, we trained a sparsely connected network that runs on the TrueNorth chip using the MNIST dataset. With a high performance network (ensemble of 64), we achieve 99.42% accuracy at 108 μJ per image, and with a high efficiency network (ensemble of 1) we achieve 92.7% accuracy at 0.268 μJ per image.