Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

TLDR

Machine‑learning performance hinges on selecting appropriate hyperparameters. The paper proposes Hyperband, a bandit‑based method to accelerate random search via adaptive resource allocation and early stopping. Hyperband treats hyperparameter search as an infinite‑armed bandit, allocating resources such as iterations or data samples to randomly sampled configurations and iteratively pruning poor performers. Hyperband achieves more than a ten‑fold speedup compared to state‑of‑the‑art Bayesian methods on diverse deep‑learning and kernel‑based tasks.

Abstract

Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation and early-stopping. We formulate hyperparameter optimization as a pure-exploration non-stochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations. We introduce a novel algorithm, Hyperband, for this framework and analyze its theoretical properties, providing several desirable guarantees. Furthermore, we compare Hyperband with popular Bayesian optimization methods on a suite of hyperparameter optimization problems. We observe that Hyperband can provide over an order-of-magnitude speedup over our competitor set on a variety of deep-learning and kernel-based learning problems.