IEEE Transactions on Parallel and Distributed Systems · 2022 · 50 citations · 31 references
ComputingComputer EngineeringComputer ScienceCompilersComputer ArchitectureParallel ComputingAccelerator ArchitectureGraphics Processing Unit ArchitectureGraphics Processing Unit ComputingGraphics Processing UnitsHardware AccelerationMatrix ComputationCompute Unified Device ArchitectureGpu Parallel AlgorithmsGpu Performance ProfilingGpu Programming ModelsMatrix MultiplicationParallel Sparse Linear AlgebraScientific Computing On GpusSparse Computation Acceleration
Tensor Cores have been an important unit to accelerate Fused Matrix Multiplication Accumulation (MMA) in all NVIDIA GPUs since Volta Architecture. To program Tensor Cores, users have to use either legacy wmma APIs or current mma APIs. Legacy wmma APIs are more easy-to-use but can only exploit limited features and power of Tensor Cores. Specifically, wmma APIs support fewer operand shapes and can not leverage the new sparse matrix multiplication feature of the newest Ampere Tensor Cores. However, the performance of current programming interface has not been well explored. Furthermore, the computation numeric behaviors of low-precision floating points (TF32, BF16, and FP16) supported by the newest Ampere Tensor Cores are also mysterious. In this paper, we explore the throughput and latency of current programming APIs. We also intuitively study the numeric behaviors of Tensor Cores MMA and profile the intermediate operations including multiplication, addition of inner product, and accumulation. All codes used in this work can be found in <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/sunlex0717/DissectingTensorCores</uri> .
31
In-Datacenter Performance Analysis of a Tensor Processing Unit
Norman P. Jouppi, Cliff Young, Nishant Patil et al. · 2017 · 4.4K citations
Analyzing CUDA workloads using a detailed GPU simulator
Ali Bakhoda, George L. Yuan, Wilson Wai Lun Fung et al. · 2009 · 1.7K citations
Fast Algorithms for Convolutional Neural Networks
Andrew Lavin, Scott Gray · 2016 · 917 citations
Paulius Micikevicius, Sharan Narang, Jonah Alben et al. · arXiv (Cornell University) · 2017 · 881 citations · Full text
Demystifying GPU microarchitecture through microbenchmarking
Henry Wong, Misel-Myrto Papadopoulou, Maryam Sadooghi-Alvandi et al. · 2010 · 447 citations