2019 · 76 citations · 24 references
ComputingComputer ScienceHigh-performance ComputingComputer ArchitectureParallel AlgorithmsParallel ComputingDeep LearningGraphics Processing Unit ArchitectureGraphics Processing Unit ComputingGraphics Processing UnitsMatrix ComputationParallel ArchitecturesCuda ProgrammingDeep Learning On GpusKernel OptimizationMatrix MultiplicationParallel Matrix MultiplicationScientific Computing AccelerationScientific Computing On GpusThread Parallelism
General matrix multiplication (GEMM) plays a paramount role in a broad range of domains such as deep learning, scientific computing, and image processing. The primary optimization method is to partition the matrix into many tiles and exploit the parallelism within and between tiles. The tiling hierarchy closely mirrors the thread hierarchy on GPUs. In practice, GPUs can fully unleash its computing power only when the matrix size is large and there are sufficient number of tiles and workload for each tile. However, in many real-world applications especially deep learning domain, the matrix size is small. To this end, prior work proposes batched GEMM to process a group of small independent GEMMs together by designing a single CUDA kernel for all of these GEMMs.
24
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
cuDNN: Efficient Primitives for Deep Learning
Sharan Chetlur, Cliff Woolley, Philippe Vandermersch et al. · arXiv (Cornell University) · 2014 · 1K citations · Full text