Concepedia

Gradient Synchronization

Gradient Synchronization is a data parallelism technique that belongs to the class of distributed optimization methods, specifically distinguished by its coordination of gradient updates across multiple processing nodes to ensure model consistency during synchronous training.

48

Publications

3.2K

Citations

164

Authors

55

Institutions

Publications per year

2017–2026

32

Parents

Publications

48

PyTorch distributed

Li Shen, Yanli Zhao, Rohan Varma et al. · Proceedings of the VLDB Endowment · 2020 · 424 citations

, , +17

TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning

Wei Wen, Cong Xu, Feng Yan et al. · arXiv (Cornell University) · 2017 · 422 citations · Full text

, , +17

TernGrad: ternary gradients to reduce communication in distributed deep learning

Wei Wen, Cong Xu, Feng Yan et al. · arXiv (Cornell University) · 2017 · 396 citations · Full text

, , +17

A generic communication scheduler for distributed DNN training acceleration

Yanghua Peng, Yibo Zhu, Yangrui Chen et al. · 2019 · 319 citations

, , +17

Hiding global synchronization latency in the preconditioned Conjugate Gradient algorithm

Pieter Ghysels, Wim Vanroose · Parallel Computing · 2013 · 155 citations

, , +11

1–5 of 48

Authors

164

  1. CY
    Cheng Yang

    Tsinghua University, China

    4 publications · 74 citations

    4

    h-index

  2. FK
    Fabian Kühn

    ETH Zurich, Switzerland

    3 publications · 73 citations

    3

    h-index

  3. TL
    Thomas Locher

    ETH Zurich, Switzerland

    3 publications · 73 citations

    3

    h-index

  4. FY
    Feng Yan

    University of Nevada, Reno, United States

    3 publications · 854 citations

    3

    h-index

  5. DL
    Dan Li

    Tsinghua University, China

    3 publications · 64 citations

    3

    h-index

1–5 of 164

Institutions

55

  1. Tsinghua University

    Beijing, China

    5 h-index

    25 publications

    486 citations

  2. Meta (Israel)

    Tel Aviv, Israel

    3 h-index

    21 publications

    5K citations

  3. Massachusetts Institute of Technology

    Cambridge, United States

    3 h-index

    4 publications

    114 citations

  4. ETH Zurich

    Zurich, Switzerland

    3 h-index

    3 publications

    73 citations

  5. University of Nevada, Reno

    Reno, United States

    3 h-index

    4 publications

    890 citations

  6. University of Illinois Urbana-Champaign

    Urbana, United States

    2 h-index

    2 publications

    46 citations

  7. Hewlett-Packard (United States)

    Palo Alto, United States

    2 h-index

    2 publications

    818 citations

  8. Amazon (United States)

    Seattle, United States

    2 h-index

    7 publications

    119 citations

1–8 of 55

Venues

  1. arXiv (Cornell University)

    Repository · 11 publications · 1.2K citations

    10

    h-index

  2. Neural Information Processing Systems

    Conference · 3 publications · 51 citations

    3

    h-index

  3. Proceedings of the VLDB Endowment

    PVLDB · 2 publications · 441 citations

    2

    h-index

  4. IBM Journal of Research and Development

    1 publications · 10 citations

    1

    h-index

  5. Theory of Computing Systems

    1 publications · 20 citations

    1

    h-index

1–5