Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent

Xiangru Lian, Ce Zhang, Huan Zhang, +3

Neural Information Processing Systems

Cluster ComputingEngineeringMachine LearningDistributed AlgorithmsNetwork Analysis +17

Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent

Xiangru Lian, Ce Zhang, Huan Zhang, +3

arXiv (Cornell University)

Cluster ComputingEngineeringMachine LearningDistributed AlgorithmsNetwork Analysis +17

Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization

Xiangru Lian, Yijun Huang, Yuncheng Li, +1

arXiv (Cornell University)

Asynchronous Parallel ImplementationsNondifferentiable OptimizationEngineeringMachine LearningNonconvex Optimization +13

Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization

Xiangru Lian, Yijun Huang, Yuncheng Li, +1

arXiv (Cornell University)

Asynchronous Parallel ImplementationsNondifferentiable OptimizationEngineeringMachine LearningNonconvex Optimization +13

D$^2$: Decentralized Training over Decentralized Data

Hanlin Tang, Xiangru Lian, Ming Yan, +2

arXiv (Cornell University)

EngineeringMachine LearningDistributed AlgorithmsMultiple WorkersDistributed Ai System +18

Staleness-aware async-SGD for distributed deep learning

Wei Zhang, Suyog Gupta, Xiangru Lian, +1

International Joint Conference on Artificial Intelligence

Convolutional Neural NetworkEngineeringMachine LearningDistributed AlgorithmsDistributed Ai System +14

DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression

Hanlin Tang, Xiangru Lian, Chen Yu, +2

arXiv (Cornell University)

Double-pass Error-compensated CompressionCluster ComputingEngineeringMachine LearningDistributed Algorithms +15

Asynchronous Decentralized Parallel Stochastic Gradient Descent

Xiangru Lian, Wei Zhang, Ce Zhang, +1

arXiv (Cornell University)

Cluster ComputingEngineeringMachine LearningDistributed Ai SystemDistributed Data Analytics +14

Staleness-aware Async-SGD for Distributed Deep Learning

Wei Zhang, Suyog Gupta, Xiangru Lian, +1

arXiv (Cornell University)

Convolutional Neural NetworkEngineeringMachine LearningDistributed AlgorithmsDistributed Ai System +14

A Comprehensive Linear Speedup Analysis for Asynchronous Stochastic Parallel Optimization from Zeroth-Order to First-Order

Xiangru Lian, Huan Zhang, Cho‐Jui Hsieh, +2

Neural Information Processing Systems

Mathematical ProgrammingNumerical AnalysisAsynchronous ParallelEngineeringParallel Processing +13