2007 · 62 citations · 13 references
ComputingComputer ScienceComputational LinguisticsNatural Language ProcessingText AnalyticsLanguage ModelingLanguage ModelsNatural Language GenerationParsingText GenerationText SummarizationAutomated Text GenerationAutomatic Mt EvaluationAutomatic Summarization EvaluationEvaluation Of Text GenerationFaithfulness In GenerationHuman Evaluation Of Machine TranslationLanguage GenerationSentence Fluency Assessment
In evaluating the output of language technology applications—MT, natural language generation, summarisation—automatic evaluation techniques generally conflate measurement of faithfulness to source content with fluency of the resulting text. In this paper we develop an automatic evaluation metric to estimate fluency alone, by examining the use of parser outputs as metrics, and show that they correlate with human judgements of generated text fluency. We then develop a machine learner based on these, and show that this performs better than the individual parser metrics, approaching a lower bound on human performance. We finally look at different language models for generating sentences, and show that while individual parser metrics can be ‘fooled ’ depending on generation method, the machine learner provides a consistent estimator of fluency. 1
13
Kishore Papineni, Salim Roukos, Todd J. Ward et al. · 2001 · 21.5K citations · Full text
Making Large-Scale SVM Learning Practical
Thorsten Joachims · Technical reports · 2006 · 4.3K citations · Full text
Magnitude Estimation of Linguistic Acceptability
Ellen Gurman Bard, D. H. Robertson, Antonella Sorace · Language · 1996 · 722 citations