2021 · 190 citations · 56 references
ComputingComputer EngineeringComputer ScienceComputer VisionMemoryNatural Language ProcessingAttention NetworksCoarse-grained Reconfigurable ArchitecturesDeep LearningHardware AccelerationMemory Hierarchy OptimizationPattern RecognitionAttentionAttention MechanismsCoarse-grained SamplingContextual Knowledge RepresentationLow-power Hardware ModelingMatrix MultiplicationMemory Footprint ReductionSparse Learning
In recent years, attention-based models have achieved impressive performance in natural language processing and computer vision applications by effectively capturing contextual knowledge from the entire sequence. However, the attention mechanism inherently contains a large number of redundant connections, imposing a heavy computational burden on model deployment. To this end, sparse attention has emerged as an attractive approach to reduce the computation and memory footprint, which involves the sampled dense-dense matrix multiplication (SDDMM) and sparse-dense matrix multiplication (SpMM) at the same time, thus requiring the hardware to eliminate zero-valued operations effectively. Existing techniques based on irregular sparse patterns or regular but coarse-grained patterns lead to low hardware efficiency or less computation saving.
56
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Adam Paszke, Sam Gross, Francisco Massa et al. · arXiv (Cornell University) · 2019 · 16.2K citations · Full text
Xiaolong Wang, Ross Girshick, Abhinav Gupta et al. · 2018 · 11.2K citations
Transformers: State-of-the-Art Natural Language Processing
Thomas Wolf, Lysandre Debut, Victor Sanh et al. · 2020 · 8K citations · Full text
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Kelvin Xu, Jimmy Ba, Ryan Kiros et al. · arXiv (Cornell University) · 2015 · 7.5K citations · Full text
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev et al. · 2016 · 6.3K citations · Full text