SGDR: Stochastic Gradient Descent with Warm Restarts

TLDR

Restart techniques are common in gradient‑free optimization for multimodal functions and are increasingly used in gradient‑based methods to accelerate convergence on ill‑conditioned problems. This paper proposes a simple warm‑restart strategy for stochastic gradient descent to enhance its anytime performance when training deep neural networks. The technique applies periodic warm restarts during SGD, adjusting learning rates to maintain momentum and improve convergence speed. The approach yields new state‑of‑the‑art results on CIFAR‑10 (3.14%) and CIFAR‑100 (16.21%), also improving performance on EEG recordings and a downsampled ImageNet dataset, with source code available online.

Abstract

Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at this https URL