2021 · 102 citations · 12 references
ComputingCommunicationComputer ScienceCluster ComputingMachine LearningParallel ComputingDeep LearningGraph AlgorithmsGraph AnalyticsGraph MiningGraph-based LearningScalable Machine LearningBandwidthBenchmark DatasetsCommunication AvoidanceCommunication Overhead ReductionGraph Neural NetworksMemory CapacityShared MemorySockets
Full-batch training on Graph Neural Networks (GNN) to learn the structure of large graphs is a critical problem that needs to scale to hundreds of compute nodes to be feasible. It is challenging due to large memory capacity and bandwidth requirements on a single compute node and high communication volumes across multiple nodes. In this paper, we present DistGNN that optimizes the well-known Deep Graph Library (DGL) for full-batch training on CPU clusters via an efficient shared memory implementation, communication reduction using a minimum vertex-cut graph partitioning algorithm and communication avoidance using a family of delayed-update algorithms. Our results on four common GNN benchmark datasets: Reddit, OGB-Products, OGB-Papers and Proteins, show up to 3.7× speed-up using a single CPU socket and up to 97× speed-up using 128 CPU sockets, respectively, over baseline DGL implementations running on a single CPU socket.
12
Proceedings of the 21st International Conference on Neural Information Processing Systems
Daphne Koller, Dale Schuurmans, Yoshua Bengio et al. · 2008 · 4.5K citations
Geometric Deep Learning: Going beyond Euclidean data
Michael M. Bronstein, Joan Bruna, Yann LeCun et al. · IEEE Signal Processing Magazine · 2017 · 3.6K citations · Full text
Grzegorz Malewicz, Matthew H. Austern, Aart J. C. Bik et al. · 2010 · 3.5K citations
George Karypis, Vipin Kumar · University of Minnesota Digital Conservancy (University of Minnesota) · 1997 · 1.2K citations · Full text