Extrapolating Learning Curves of Deep Neural Networks

Abstract

• Representative power increased by convex combination of individual models: • f x = wifi(x|θi) k i=1 + e with e ~ N 0, σ 2 and wi = 1 k i=1 • Model uncertainty captured by MCMC • The prior encoded monotonicity assumption of each of the models • We obtained S = 100000 samples from 100 parallel chains of length 1500 with a burn-in of 500 • Let ξ be the model’s parameters w1, ... , wk , θ1, ... , θk , σ 2 • Probability of improving over current best parameter setting: