arXiv (Cornell University) · 2020 · 605 citations · 64 references
Open access
ComputingComputer ScienceComputational LinguisticsInformation RetrievalNatural Language ProcessingLanguage ModelingMachine TranslationNatural Language GenerationText GenerationWord EmbeddingsAutomatic Mt EvaluationContextual EmbeddingsContextual Word EmbeddingsEvaluation Of Text GenerationHuman Evaluation Of Machine TranslationImage CaptioningMetricsParaphrase Generation
BERTScore computes a token‑wise similarity score between candidate and reference sentences, analogous to common metrics. The paper proposes BERTScore, an automatic evaluation metric for text generation. BERTScore uses contextual embeddings instead of exact matches to compute token similarity and is evaluated on outputs from 363 machine translation and image captioning systems. BERTScore correlates better with human judgments, improves model selection, and is more robust to challenging examples than existing metrics.
We propose BERTScore, an automatic evaluation metric for text generation. Analogously to common metrics, BERTScore computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTScore correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task and show that BERTScore is more robust to challenging examples compared to existing metrics.
64
Kishore Papineni, Salim Roukos, Todd J. Ward et al. · 2001 · 21.5K citations · Full text
Distributed Representations of Words and Phrases and their Compositionality
Tomáš Mikolov, Ilya Sutskever, Kai Chen et al. · arXiv (Cornell University) · 2013 · 18.1K citations · Full text
ROUGE: A Package for Automatic Evaluation of Summaries
Chin-Yew Lin · 2004 · 8.3K citations