ZeRO: Memory optimizations Toward Training Trillion Parameter Models

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, +1

EngineeringMachine LearningModel TuningComputer ArchitectureData Science +14

DeepSpeed

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, +1

Large Language ModelsNatural Language ProcessingLarge Ai ModelLlm Fine-tuningOpen Source Library +13

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

Shaden Smith, Mostofa Patwary, Brandon Norick, +17

arXiv (Cornell University)

Artificial IntelligenceLlm Fine-tuningMegatron-turing Nlg 530BMachine LearningEngineering +19

OpenFold: retraining AlphaFold2 yields new insights into its learning mechanisms and capacity for generalization

Gustaf Ahdritz, Nazim Bouatta, Christina Floristean, +31

Nature Methods

DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale

Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, +7

Convolutional Neural NetworkEngineeringMachine LearningComputer ArchitectureDeepspeed- Inference +17

ZeRO-infinity

Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, +2

Cluster ComputingGpu ArchitectureEngineeringMachine LearningData Science +13

The Cilkview scalability analyzer

Yuxiong He, Charles E. Leiserson, William M. Leiserson

Cluster ComputingEngineeringCilkview Scalability AnalyzerComputer ArchitectureSoftware Engineering +19

Performance Modeling and Scalability Optimization of Distributed Deep Learning Systems

Feng Yan, Olatunji Ruwase, Yuxiong He, +1

Cluster ComputingConvolutional Neural NetworkEngineeringMachine LearningDistributed Ai System +19

Learning Intrinsic Sparse Structures within Long Short-Term Memory

Wei Wen, Yuxiong He, Samyam Rajbhandari, +6

arXiv (Cornell University)

Structured PredictionGeometric LearningEngineeringMachine LearningGroup Lasso Regularization +17

Provably-Efficient Job Scheduling for Energy and Fairness in Geographically Distributed Data Centers

Shaolei Ren, Yuxiong He, Fei Xu

Cluster ComputingEngineeringEnergy EfficiencyAffine CombinationDesirable Tradeoff +17