2011 · 128 citations · 38 references
ComputingBiostatisticsMachine LearningStatistical LearningStatistical ModelingBayesian InferenceGaussian ProcessesLikelihood EstimationMarkov Chain Monte CarloMaximum Likelihood EstimationStatistical InferenceConvergenceGaussian Process RegressionMarkov Chain Monte Carlo EstimationPredictive Model PerformancePredictive PerformanceRobust RegressionVariational BayesVariational Bayes Inference
This paper considers the robust and efficient implementation of Gaussian process regression with a Student-t observation model, which has a non-log-concave likelihood. The challenge with the Student-t model is the analytically intractable inference which is why several approximative methods have been proposed. Expectation propagation (EP) has been found to be a very accurate method in many empirical studies but the convergence of EP is known to be problematic with models containing non-log-concave site functions. In this paper we illustrate the situations where standard EP fails to converge and review different modifications and alternative algorithms for improving the convergence. We demonstrate that convergence problems may occur during the type-II maximum a posteriori (MAP) estimation of the hyperparameters and show that standard EP may not converge in the MAP values with some difficult data sets. We present a robust implementation which relies primarily on parallel EP updates and uses a moment-matching-based double-loop algorithm with adaptively selected step size in difficult cases. The predictive performance of EP is compared with Laplace, variational Bayes, and Markov chain Monte Carlo approximations. Keywords: Gaussian process, robust regression, Student-t distribution, approximate inference, expectation propagation
38
Pattern Recognition and Machine Learning
Journal of Electronic Imaging · 2007 · 22.1K citations
Multivariate Adaptive Regression Splines
Jerome H. Friedman · The Annals of Statistics · 1991 · 8.1K citations · Full text
Practical Markov Chain Monte Carlo
Charles J. Geyer · Statistical Science · 1992 · 2.1K citations
Engineering, Mathematics, Statistics +16