2018 · 142 citations · 34 references
Language StudiesComputer ScienceComputational LinguisticsNatural Language ProcessingText AnalyticsCorpus CompilationGrammarSentence StructureText GenerationText SimplificationText SummarizationAutomatic Text Simplification EvaluationCorrelation AnalysisHuman Evaluation Of SimplificationParaphrase GenerationParaphrase RecognitionSentence SimplificationSimplificationStructural Simplification
BLEU is widely used as an informative metric for text-to-text generation, including text simplification, which encompasses both lexical and structural changes. This study demonstrates that BLEU is unsuitable for evaluating sentence splitting, the primary structural simplification operation. The authors compiled a gold-standard sentence-splitting corpus with multiple structural paraphrases and conducted a correlation analysis with human judgments. They found that BLEU shows low or no correlation with grammaticality and meaning preservation in sentence splitting and often negatively correlates with simplicity, penalizing simpler sentences.
BLEU is widely considered to be an informative metric for text-to-text generation, including Text Simplification (TS). TS includes both lexical and structural aspects. In this paper we show that BLEU is not suitable for the evaluation of sentence splitting, the major structural simplification operation. We manually compiled a sentence splitting gold standard corpus containing multiple structural paraphrases, and performed a correlation analysis with human judgments. We find low or no correlation between BLEU and the grammaticality and meaning preservation parameters where sentence splitting is involved. Moreover, BLEU often negatively correlates with simplicity, essentially penalizing simpler sentences.
34
Kishore Papineni, Salim Roukos, Todd J. Ward et al. · 2001 · 21.5K citations · Full text
NLTK: The Natural Language Toolkit
Edward Loper, Steven Bird · ArXiv.org · 2002 · 1.9K citations · Full text