2004 · 1.1K citations · 9 references
ComputingComputer ScienceAlgorithmsMachine LearningStatistical LearningGradient-based OptimizationNeural NetworksNumerical AlgorithmsPattern RecognitionStochastic OptimizationSupport Vector MachinesLeast Squares ProblemsLinear PredictionLogistic RegressionMachine Learning Prediction ModelsRegularizationRegularized RegressionStatistical Learning TheoryStochastic Gradient Descent
Linear prediction methods such as least squares, logistic regression, and support vector machines are widely used in statistics and machine learning, are efficient and simple to implement, and are related to online algorithms like the perceptron. The study investigates stochastic gradient descent algorithms applied to regularized linear prediction methods. The authors analyze SGD on regularized linear prediction models, derive numerical convergence rates, discuss implications, and plan experiments on text data to illustrate the theoretical findings. The analysis yields numerical convergence rates for SGD on regularized linear prediction, underscoring its practical effectiveness.
Linear prediction methods, such as least squares for regression, logistic regression and support vector machines for classification, have been extensively used in statistics and machine learning. In this paper, we study stochastic gradient descent (SGD) algorithms on regularized forms of linear prediction methods. This class of methods, related to online algorithms such as perceptron, are both efficient and very simple to implement. We obtain numerical rate of convergence for such algorithms, and discuss its implications. Experiments on text data will be provided to demonstrate numerical and statistical consequences of our theoretical findings.
9
Discriminative training methods for hidden Markov models
Michael Collins · 2002 · 1.9K citations · Full text
Large margin classification using the perceptron algorithm
Yoav Freund, Robert E. Schapire · 1998 · 1.1K citations