Publication | Open Access
Direct Training for Spiking Neural Networks: Faster, Larger, Better
106
Citations
0
References
2018
Year
EngineeringMachine LearningSocial SciencesDirect TrainingSparse Neural NetworkSpiking Neural NetworksNeuromorphic EngineeringNeurocomputersComputer EngineeringNeuromorphic ComputingComputer ScienceNeural NetworksDeep LearningNeural Architecture SearchDeep SnnsDeep Neural NetworksComputational NeuroscienceNon-spiking DatasetsNeuroscienceBrain-like Computing
Spiking neural networks promise energy‑efficient neuromorphic computing, yet they have lagged behind artificial neural networks because of ineffective learning algorithms and limited programming frameworks. This study aims to overcome these limitations by introducing a neuron‑normalization technique and a direct learning algorithm for deep SNNs. The authors implement these methods in PyTorch, narrowing the rate‑coding window and converting the leaky integrate‑and‑fire model into an explicit iterative form to enable large‑scale training. The resulting approach trains deep SNNs with tens‑of‑times speedup, achieves superior accuracy on neuromorphic datasets, matches ANN performance on non‑spiking data, and represents the first high‑performance direct training of deep SNNs on CIFAR‑10.
Spiking neural networks (SNNs) that enables energy efficient implementation on emerging neuromorphic hardware are gaining more attention. Yet now, SNNs have not shown competitive performance compared with artificial neural networks (ANNs), due to the lack of effective learning algorithms and efficient programming frameworks. We address this issue from two aspects: (1) We propose a neuron normalization technique to adjust the neural selectivity and develop a direct learning algorithm for deep SNNs. (2) Via narrowing the rate coding window and converting the leaky integrate-and-fire (LIF) model into an explicitly iterative version, we present a Pytorch-based implementation method towards the training of large-scale SNNs. In this way, we are able to train deep SNNs with tens of times speedup. As a result, we achieve significantly better accuracy than the reported works on neuromorphic datasets (N-MNIST and DVSCIFAR10), and comparable accuracy as existing ANNs and pre-trained SNNs on non-spiking datasets (CIFAR10). To our best knowledge, this is the first work that demonstrates direct training of deep SNNs with high performance on CIFAR10, and the efficient implementation provides a new way to explore the potential of SNNs.