Research Showcase @ Carnegie Mellon University (Carnegie Mellon University) · 2018 · 99 citations · 18 references
ComputingComputer ScienceMachine LearningOptimizationStatistical ModelingBias-variance TradeoffConvex OptimizationGradient-based OptimizationOptimization For Machine LearningStochastic OptimizationVariational InferenceConvergenceGradient NoiseLogistic RegressionStochastic Gradient DescentStochastic Variational InferenceStochastic Variational OptimizationVariance Reduction OptimizationVariance Reduction Techniques
Stochastic gradient optimization trains machine learning models using noisy gradients from random samples, but high variance can cause slow convergence and poor performance. This work proposes a general control‑variate method to reduce variance in stochastic gradient optimization. The method constructs control variates from pre‑computed or online‑estimated low‑order moments and applies them to convex logistic regression MAP and non‑convex LDA variational inference. Experiments on both problems demonstrate faster convergence and superior performance compared to the classical stochastic gradient method.
Stochastic gradient optimization is a class of widely used algorithms for training machine learning models. To optimize an objective, it uses the noisy gradient computed from the random data samples instead of the true gradient computed from the entire dataset. However, when the variance of the noisy gradient is large, the algorithm might spend much time bouncing around, leading to slower convergence and worse performance. In this paper, we develop a general approach of using control variate for variance reduction in stochastic gradient. Data statistics such as low-order moments (pre-computed or estimated online) is used to form the control variate. We demonstrate how to construct the control variate for two practical problems using stochastic gradient optimization. One is convex—the MAP estimation for logistic regression, and the other is non-convex—stochastic variational inference for latent Dirichlet allocation. On both problems, our approach shows faster convergence and better performance than the classical approach.
18
Pattern Recognition and Machine Learning
Journal of Electronic Imaging · 2007 · 22.1K citations
Algorithms for Non-negative Matrix Factorization
Daniel D. Lee, H. Sebastian Seung · 2000 · 5.5K citations
ALGORITHMS FOR NON-NEGATIVE MATRIX FACTORIZATION
D Seung, Lin-Wen Lee · 2001 · 4.8K citations
An Introduction to Variational Methods for Graphical Models
Michael I. Jordan, Zoubin Ghahramani, Tommi Jaakkola et al. · Machine Learning · 1999 · 3.8K citations · Full text