arXiv (Cornell University) · 2018 · 18 citations · 0 references
ComputingComputer ScienceAlgorithmsMachine LearningStochastic ProcessesAdaptive LearningBias-variance TradeoffDeep LearningGradient-based OptimizationNeural NetworksReinforcement LearningAdaptive OptimizationAdaptive RateConvergenceCorrelationGradient DescentOptimization AlgorithmsSecond-order MomentStochastic Gradient Descent
Adam is shown not being able to converge to the optimal solution in certain\ncases. Researchers recently propose several algorithms to avoid the issue of\nnon-convergence of Adam, but their efficiency turns out to be unsatisfactory in\npractice. In this paper, we provide new insight into the non-convergence issue\nof Adam as well as other adaptive learning rate methods. We argue that there\nexists an inappropriate correlation between gradient $g_t$ and the\nsecond-moment term $v_t$ in Adam ($t$ is the timestep), which results in that a\nlarge gradient is likely to have small step size while a small gradient may\nhave a large step size. We demonstrate that such biased step sizes are the\nfundamental cause of non-convergence of Adam, and we further prove that\ndecorrelating $v_t$ and $g_t$ will lead to unbiased step size for each\ngradient, thus solving the non-convergence problem of Adam. Finally, we propose\nAdaShift, a novel adaptive learning rate method that decorrelates $v_t$ and\n$g_t$ by temporal shifting, i.e., using temporally shifted gradient $g_{t-n}$\nto calculate $v_t$. The experiment results demonstrate that AdaShift is able to\naddress the non-convergence issue of Adam, while still maintaining a\ncompetitive performance with Adam in terms of both training speed and\ngeneralization.\n