2023 · 80 citations · 28 references
ComputingComputer ScienceArtificial IntelligenceKnowledge RepresentationMachine LearningDeep LearningNeural ModelingNeural NetworksSpiking Neural NetworksDeep Spiking Neural NetworksEnergy-efficient ComputationGradient DescentKnowledge DistillationSpiking ActivitySpiking Neural Network ModelsSpiking Neuron ModelsTraining
Spiking neural networks (SNNs) are well-known as brain-inspired models with high computing efficiency, due to a key component that they utilize spikes as information units, close to the biological neural systems. Although spiking based models are energy efficient by taking advantage of discrete spike signals, their performance is limited by current network structures and their training methods. As discrete signals, typical SNNs cannot apply the gradient descent rules directly into parameter adjustment as artificial neural networks (ANNs). Aiming at this limitation, here we propose a novel method of constructing deep SNN models with knowledge distillation (KD) that uses ANN as the teacher model and SNN as the student model. Through the ANN-SNN joint training algorithm, the student SNN model can learn rich feature information from the teacher ANN model through the KD method, yet it avoids training SNN from scratch when communicating with non-differentiable spikes. Our method can not only build a more efficient deep spiking structure feasibly and reasonably but use few time steps to train the whole model compared to direct training or ANN to SNN methods. More importantly, it has a superb ability of noise immunity for various types of artificial noises and natural signals. The proposed novel method provides efficient ways to improve the performance of SNN through constructing deeper structures in a high-throughput fashion, with potential usage for light and efficient brain-inspired computing of practical scenarios.
28
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
Distilling the Knowledge in a Neural Network
Geoffrey E. Hinton, Oriol Vinyals · arXiv (Cornell University) · 2015 · 14K citations · Full text
FitNets: Hints for Thin Deep Nets
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou et al. · arXiv (Cornell University) · 2014 · 2K citations · Full text
Education, Educational Technology, Artificial Intelligence In Education +8
Emre Neftci, Hesham Mostafa, Friedemann Zenke · IEEE Signal Processing Magazine · 2019 · 1.4K citations
Life Sciences, Computational Engineering, Computational Neuroscience +12
Spatio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks
Yujie Wu, Lei Deng, Guoqi Li et al. · Frontiers in Neuroscience · 2018 · 1.1K citations · Full text