DiSCO: Distributed Optimization for Self-Concordant Empirical Loss

Abstract

We propose a new distributed algorithm for em-pirical risk minimization in machine learning. The algorithm is based on an inexact damped Newton method, where the inexact Newton steps are computed by a distributed preconditioned conjugate gradient method. We analyze its iter-ation complexity and communication efficiency for minimizing self-concordant empirical loss functions, and discuss the results for distributed ridge regression, logistic regression and binary classification with a smoothed hinge loss. In a standard setting for supervised learning, where the n data points are i.i.d. sampled and when the regularization parameter scales as 1/ n, we show that the proposed algorithm is communica-tion efficient: the required round of communica-tion does not increase with the sample size n, and only grows slowly with the number of machines. 1.