Frontiers in Neuroscience · 2024 · 12 citations · 65 references
ComputingComputer ScienceNeuroscienceArtificial IntelligenceMachine LearningNeural EngineeringDeep LearningNeural Circuit ModelingNeural NetworksSpiking Neural NetworksEnergy ConsumptionEnergy Consumption OptimizationEvolutionary SearchNeural Architecture SearchPareto Optimal SolutionSpiking Network ArchitecturesSpiking Neural Network ModelsTransformer Models
Introduction: The integration of self-attention mechanisms into Spiking Neural Networks (SNNs) has garnered considerable interest in the realm of advanced deep learning, primarily due to their biological properties. Recent advancements in SNN architecture, such as Spikformer, have demonstrated promising outcomes. However, we observe that Spikformer may exhibit excessive energy consumption, potentially attributable to redundant channels and blocks. Methods: To mitigate this issue, we propose a one-shot Spiking Transformer Architecture Search method, namely Auto-Spikformer. Auto-Spikformer extends the search space to include both transformer architecture and SNN inner parameters. We train and search the supernet based on weight entanglement, evolutionary search, and the proposed Discrete Spiking Parameters Search (DSPS) methods. Benefiting from these methods, the performance of subnets with weights inherited from the supernet without even retraining is comparable to the original Spikformer. Moreover, we propose a new fitness function aiming to find a Pareto optimal combination balancing energy consumption and accuracy. Results and discussion: Our experimental results demonstrate the effectiveness of Auto-Spikformer, which outperforms the original Spikformer and most CNN or ViT models with even fewer parameters and lower energy consumption.
65
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · 2023 · 76.3K citations · Full text
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov et al. · arXiv (Cornell University) · 2020 · 21.6K citations · Full text
Advances in neural information processing systems 7
Computers & Mathematics with Applications · 1996 · 14.4K citations
Backpropagation through time: what it does and how to do it
Paul J. Werbos · Proceedings of the IEEE · 1990 · 4.9K citations