AdaShift: Decorrelation and Convergence of Adaptive Learning Rate\n Methods

Zhiming Zhou, Qingru Zhang, Guansong Lu, Hongwei Wang, Weinan Zhang, Yong Yu

arXiv (Cornell University) · 2018 · 18 citations · 0 references

DOIFull text

Open access

Concepts

Abstract

Adam is shown not being able to converge to the optimal solution in certain\ncases. Researchers recently propose several algorithms to avoid the issue of\nnon-convergence of Adam, but their efficiency turns out to be unsatisfactory in\npractice. In this paper, we provide new insight into the non-convergence issue\nof Adam as well as other adaptive learning rate methods. We argue that there\nexists an inappropriate correlation between gradient $g_t$ and the\nsecond-moment term $v_t$ in Adam ($t$ is the timestep), which results in that a\nlarge gradient is likely to have small step size while a small gradient may\nhave a large step size. We demonstrate that such biased step sizes are the\nfundamental cause of non-convergence of Adam, and we further prove that\ndecorrelating $v_t$ and $g_t$ will lead to unbiased step size for each\ngradient, thus solving the non-convergence problem of Adam. Finally, we propose\nAdaShift, a novel adaptive learning rate method that decorrelates $v_t$ and\n$g_t$ by temporal shifting, i.e., using temporally shifted gradient $g_{t-n}$\nto calculate $v_t$. The experiment results demonstrate that AdaShift is able to\naddress the non-convergence issue of Adam, while still maintaining a\ncompetitive performance with Adam in terms of both training speed and\ngeneralization.\n