2018 · 68 citations · 38 references
ComputingComputer EngineeringComputer ScienceHardware-software Co-designInterconnection NetworksParallel ComputingGraphics Processing Unit ArchitectureGraphics Processing Unit ComputingGraphics Processing UnitsHardware AccelerationInterconnect ArchitectureMultistage Interconnection NetworksDeep Learning On GpusEmpirical EvaluationGpu ArchitectureGpu InterconnectsMulti-gpu ArchitecturesMulti-gpu ComputingMulti-gpu SystemsPerformance Obstacles
High performance multi-GPU computing becomes an inevitable trend due to the ever-increasing demand on computation capability in emerging domains such as deep learning, big data and planet-scale applications. However, the lack of deep understanding on how modern GPUs can be connected and the actual impact of state-of-the-art interconnect on multiGPU application performance becomes a hurdle. Additionally, the absence of a practical multi-GPU benchmark suite poses further obstacles for conducting research in multi-GPU era. In this paper, we fill the gap by proposing a multi-GPU benchmark suite named Tartan, which contains microbenchmarks, scale-up and scale-out applications. We then apply Tartan to evaluate the four latest types of modern GPU interconnects, i.e., PCI-e, NVLink-V1, NVLink-V2 and InfiniBand with GPUDirect-RDMA from two recently released NVIDIA super AI platforms as well as ORNL's exascale prototype system. Based on empirical evaluation, we observe four new types of NUMA effects: three types are triggered by NVLink's topology, connectivity and routing, while one type is caused by PCI-e (i.e., anti-locality). They are very important for performance tuning in multi-GPU environment. Our evaluation results show that, unless the current CPU-GPU master-slave programming model can be replaced, it is difficult for scale-up multi-GPU applications to really benefit from faster intra-node interconnects such as NVLinks; while for inter-node scale-out applications, although interconnect is more crucial to the overall performance, GPUDirect-RDMA appears to be not always the optimal choice. The Tartan benchmark suite including the microbenchmarks are opensource and available athttp://github.com/uuudown/Tartan.
38
Rodinia: A benchmark suite for heterogeneous computing
Shuai Che, Michael Boyer, Jiayuan Meng et al. · 2009 · 3K citations
A Proof for the Queuing Formula: <i>L</i> = λ<i>W</i>
John D. C. Little · Operations Research · 1961 · 2.7K citations
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Priya Goyal, Piotr Dollár, Ross Girshick et al. · arXiv (Cornell University) · 2017 · 2.6K citations · Full text
Analyzing CUDA workloads using a detailed GPU simulator
Ali Bakhoda, George L. Yuan, Wilson Wai Lun Fung et al. · 2009 · 1.7K citations