arXiv (Cornell University) · 2021 · 18 citations · 33 references
ComputingComputer ScienceHigh-performance ComputingComputer ArchitectureMachine LearningParallel ComputingAccelerator ArchitectureCompute Resource ManagementDeep LearningHardware AccelerationKernel OffloadingScalable Machine LearningConvergenceCpu OffloadingDeep Learning On GpusEfficiencyMemory Footprint ReductionTransformer Models
Recent expeditious developments in deep learning algorithms, distributed training, and even hardware design for large models have enabled training extreme-scale models, say GPT-3 and Switch Transformer possessing hundreds of billions or even trillions of parameters. However, under limited resources, extreme-scale model training that requires enormous amounts of computes and memory footprint suffers from frustratingly low efficiency in model convergence. In this paper, we propose a simple training strategy called "Pseudo-to-Real" for high-memory-footprint-required large models. Pseudo-to-Real is compatible with large models with architecture of sequential layers. We demonstrate a practice of pretraining unprecedented 10-trillion-parameter model, an order of magnitude larger than the state-of-the-art, on solely 512 GPUs within 10 days. Besides demonstrating the application of Pseudo-to-Real, we also provide a technique, Granular CPU offloading, to manage CPU memory for training large model and maintain high GPU utilities. Fast training of extreme-scale models on a decent amount of resources can bring much smaller carbon footprint and contribute to greener AI.
33
Decoupled Weight Decay Regularization
Ilya Loshchilov, Frank Hutter · arXiv (Cornell University) · 2017 · 9.1K citations · Full text
On the Opportunities and Risks of Foundation Models
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli et al. · arXiv (Cornell University) · 2021 · 2.2K citations · Full text
Yukun Zhu, Ryan Kiros, Rich Zemel et al. · 2015 · 2.1K citations · Full text
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
Linting Xue, Noah Constant, Adam P. Roberts et al. · 2021 · 1.6K citations · Full text