Bayesian Methods for Mixtures of Experts

Abstract

We present a Bayesian framework for inferring the parameters of a mixture of experts model based on ensemble learning by variational free energy minimisation. The Bayesian approach avoids the over-fitting and noise level under-estimation problems of traditional maximum likelihood inference. We demonstrate these methods on artificial problems and sunspot time series prediction. INTRODUCTION The task of estimating the parameters of adaptive models such as artificial neural networks using Maximum Likelihood (ML) is well documented eg. Geman, Bienenstock & Doursat (1992). ML estimates typically lead to models with high variance, a process known as "over-fitting". ML also yields over-confident predictions; in regression problems for example, ML underestimates the noise level. This problem is particularly dominant in models where the ratio of the number of data points in the training set to the number of parameters in the model is low. In this paper we consider inference of the parameters o...