Improved Adam Optimizer for Deep Neural Networks

TLDR

Adaptive optimizers such as Adam and RMSprop outperform SGD in some settings but frequently yield worse generalization for deep neural networks. This study identifies why Adam generalizes poorly compared to SGD and proposes a variant to close the gap, aiming to illuminate why certain optimizers generalize better. The authors introduce normalized direction‑preserving Adam (ND‑Adam), which precisely controls update direction and step size, and further regularize softmax logits to enhance classification performance. ND‑Adam achieves markedly better generalization than standard Adam, and softmax‑logit regularization further boosts performance on classification tasks.

Abstract

Adaptive optimization algorithms, such as Adam and RMSprop, have witnessed better optimization performance than stochastic gradient descent (SGD) in some scenarios. However, recent studies show that they often lead to worse generalization performance than SGD, especially for training deep neural networks (DNNs). In this work, we identify the reasons that Adam generalizes worse than SGD, and develop a variant of Adam to eliminate the generalization gap. The proposed method, normalized direction-preserving Adam (ND-Adam), enables more precise control of the direction and step size for updating weight vectors, leading to significantly improved generalization performance. Following a similar rationale, we further improve the generalization performance in classification tasks by regularizing the softmax logits. By bridging the gap between SGD and Adam, we also hope to shed light on why certain optimization algorithms generalize better than others.