Model Parallelism

Model Parallelism is a distributed computing strategy in scalable machine learning wherein the parameters and computations of a single machine learning model are partitioned across multiple devices or nodes, enabling concurrent execution of different model segments during training or inference.

231

Publications

17K

Citations

Authors

216

Institutions

Publications per year

2017–2026

159

Parents

Machine Learning For Big DataScalable Machine LearningVine Architecture

Publications

231

Building high-level features using large scale unsupervised learning

Quoc V. Le · 2013 · 1.7K citations

Computing, Computer Science, Computer Vision +14

1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs

Frank Seide, Hao Fu, Jasha Droppo et al. · 2014 · 889 citations

Computing, Computer Engineering, Computer Science +17

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Mohammad Shoeybi, Mostofa Patwary, Raul Puri et al. · arXiv (Cornell University) · 2019 · 826 citations · Full text

Computing, Computer Science, High-performance Computing +16

ZeRO: Memory optimizations Toward Training Trillion Parameter Models

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase et al. · 2020 · 741 citations

Computing, Computer Engineering, Computer Science +16

Building high-level features using large scale unsupervised learning

Marc’Aurelio Ranzato, Rajat Monga, Matthieu Devin et al. · International Conference on Machine Learning · 2012 · 669 citations

Computing, Computer Science, Computer Vision +14

1–5 of 231

Authors

OR
Olatunji Ruwase
Carnegie Mellon University, United States
6 publications · 1.2K citations
6
h-index
EP
Eric P. Xing
Carnegie Mellon University, United States
6 publications · 703 citations
6
h-index
DL
Dongsheng Li
National University of Defense Technology, China
6 publications · 192 citations
6
h-index
YY
Yang You
National University of Singapore, Singapore
6 publications · 251 citations
6
h-index
YH
Yuxiong He
Microsoft (United States), United States
6 publications · 1.2K citations
6
h-index

1–5 of 1K

Institutions

216

1–8 of 216

Venues

arXiv (Cornell University)
Repository · 62 publications · 4.5K citations
31
h-index
IEEE Transactions on Parallel and Distributed Systems
TPDS · 6 publications · 245 citations
6
h-index
Proceedings of the VLDB Endowment
PVLDB · 6 publications · 360 citations
6
h-index
IEEE Transactions on Pattern Analysis and Machine Intelligence
TPAMI · 3 publications · 480 citations
3
h-index
Proceedings of the AAAI Conference on Artificial Intelligence
Conference · 2 publications · 136 citations
2
h-index

1–5