Empirical Methods in Natural Language Processing · 2004 · 37 citations · 12 references
ComputingComputer ScienceComputational LinguisticsMachine LearningNatural Language ProcessingFeature SelectionLanguage ModelingLanguage ModelsOptimization For Machine LearningRepresentation LearningText Feature EngineeringEntropy MaximizationEntropy MinimizationL1 RegularizationLikelihood MaximizationMathematical JustificationMaximum Entropy EstimationMaximum Entropy ModelingRegularization
We present an approach to bounded constraintrelaxation for entropy maximization that corresponds to using a double-exponential prior or `1 regularizer in likelihood maximization for log-linear models. We show that a combined incremental feature selection and regularization method can be established for maximum entropy modeling by a natural incorporation of the regularizer into gradientbased feature selection, following Perkins et al. (2003). This provides an efficient alternative to standard `1 regularization on the full feature set, and a mathematical justification for thresholding techniques used in likelihood-based feature selection. Also, we motivate an extension to n-best feature selection for linguistic features sets with moderate redundancy, and present experimental results showing its advantage over `0, 1-best `1, `2 regularization and over standard incremental feature selection for the task of maximum-entropy parsing.1
12