arXiv (Cornell University) · 2020 · 24 citations · 7 references
ComputingComputer ScienceEnsemble PerformanceMachine LearningParallel ComputingDeep LearningEnsemble LearningEnsemble MethodsEnsemble ModelingOptimization For Machine LearningScalable Machine LearningAccuracyDeep EnsemblesEnsemble Learning TheoryEnsemble-based PredictionMulti-model EnsemblesScalable Model Evaluation
Ensembling is a simple and popular technique for boosting evaluation performance by training multiple models (e.g., with different initializations) and aggregating their predictions. This approach is commonly reserved for the largest models, as it is commonly held that increasing the model size provides a more substantial reduction in error than ensembling smaller models. However, we show results from experiments on CIFAR-10 and ImageNet that ensembles can outperform single models with both higher accuracy and requiring fewer total FLOPs to compute, even when those individual models' weights and hyperparameters are highly optimized. Furthermore, this gap in improvement widens as models become large. This presents an interesting observation that output diversity in ensembling can often be more efficient than training larger models, especially when the models approach the size of what their dataset can foster. Instead of using the common practice of tuning a single large model, one can use ensembles as a more flexible trade-off between a model's inference speed and accuracy. This also potentially eases hardware design, e.g., an easier way to parallelize the model across multiple workers for real-time or distributed inference.
7
ImageNet Large Scale Visual Recognition Challenge
Olga Russakovsky, Jia Deng, Hao Su et al. · International Journal of Computer Vision · 2015 · 40.2K citations
Leo Breiman · Machine Learning · 1996 · 17K citations · Full text
Distilling the Knowledge in a Neural Network
Geoffrey E. Hinton, Oriol Vinyals · arXiv (Cornell University) · 2015 · 14K citations · Full text
Sergey Zagoruyko, Nikos Komodakis · arXiv (Cornell University) · 2016 · 1.9K citations · Full text