arXiv (Cornell University) · 2016 · 36 citations · 6 references
ComputingComputer ScienceMachine LearningOptimizationStatistical LearningBayesian InferenceBayesian LearningEnsemble LearningEnsemble MethodsOptimization For Machine LearningParameter OptimizationBayesian OptimizationEnsemble Learning TheoryGreedy HeuristicsHyperparameter Optimization
In this paper, we bridge the gap between hyperparameter optimization and ensemble learning by performing Bayesian optimization of an ensemble with regards to its hyperparameters. Our method consists in building a fixed-size ensemble, optimizing the configuration of one classifier of the ensemble at each iteration of the hyperparameter optimization algorithm, taking into consideration the interaction with the other models when evaluating potential performances. We also consider the case where the ensemble is to be reconstructed at the end of the hyperparameter optimization phase, through a greedy selection over the pool of models generated during the optimization. We study the performance of our proposed method on three different hyperparameter spaces, showing that our approach is better than both the best single model and a greedy ensemble construction over the models produced by a standard Bayesian optimization.
6
Random search for hyper-parameter optimization
James Bergstra, Yoshua Bengio · 2012 · 7.9K citations
Ensemble selection from libraries of models
Rich Caruana, Alexandru Niculescu-Mizil, G. Crew et al. · 2004 · 844 citations
Boosting: foundations and algorithms
Choice Reviews Online · 2012 · 534 citations
Scalable Bayesian Optimization Using Deep Neural Networks
Jasper Snoek, Oren Rippel, Kevin Swersky et al. · arXiv (Cornell University) · 2015 · 376 citations · Full text