Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

TLDR

Training large transformer models advances NLP state of the art, but memory constraints make very large models difficult to train. The study presents techniques for training very large transformer models and demonstrates that 8.3‑billion‑parameter GPT‑2‑like and 3.9‑billion‑parameter BERT‑like models can further advance SOTA. The authors implement an intra‑layer model‑parallel approach that requires only a few communication operations in native PyTorch, is orthogonal to pipeline parallelism, and enables training transformer models up to 8.3 billion parameters on 512 GPUs. The approach achieves 15.1 PetaFLOPs with 76 % scaling efficiency, and the 8.3‑billion‑parameter GPT‑2‑like model attains SOTA perplexity on WikiText103 and accuracy on LAMBADA, while the 3.9‑billion‑parameter BERT‑like model reaches SOTA accuracy on RACE, with careful layer‑norm placement proving critical for performance.

Abstract

Recent work in language modeling demonstrates that training large transformer models advances the state of the art in Natural Language Processing applications. However, very large models can be quite difficult to train due to memory constraints. In this work, we present our techniques for training very large transformer models and implement a simple, efficient intra-layer model parallel approach that enables training transformer models with billions of parameters. Our approach does not require a new compiler or library changes, is orthogonal and complimentary to pipeline model parallelism, and can be fully implemented with the insertion of a few communication operations in native PyTorch. We illustrate this approach by converging transformer based models up to 8.3 billion parameters using 512 GPUs. We sustain 15.1 PetaFLOPs across the entire application with 76% scaling efficiency when compared to a strong single GPU baseline that sustains 39 TeraFLOPs, which is 30% of peak FLOPs. To demonstrate that large language models can further advance the state of the art (SOTA), we train an 8.3 billion parameter transformer language model similar to GPT-2 and a 3.9 billion parameter model similar to BERT. We show that careful attention to the placement of layer normalization in BERT-like models is critical to achieving increased performance as the model size grows. Using the GPT-2 model we achieve SOTA results on the WikiText103 (10.8 compared to SOTA perplexity of 15.8) and LAMBADA (66.5% compared to SOTA accuracy of 63.2%) datasets. Our BERT model achieves SOTA results on the RACE dataset (90.9% compared to SOTA accuracy of 89.4%).