2018 · 91 citations · 26 references
ComputingComputer ScienceTranslation StudiesCorpus-based Translation StudiesNatural Language ProcessingPerformance MeasurementMachine TranslationTranslation Quality AssessmentTranslation TechnologyAutomated Translation EvaluationAutomatic Mt EvaluationHuman Evaluation Of Machine TranslationMachine Translation Quality EvaluationMetricsTranslation Quality BenchmarkingTranslation Quality MetricsTranslation Quality Scoring
This paper presents the results of the WMT18 Metrics Shared Task. We asked participants of this task to score the outputs of the MT systems involved in the WMT18 News Translation Task with automatic metrics. We collected scores of 10 metrics and 8 research groups. In addition to that, we computed scores of 8 standard metrics (BLEU, SentBLEU, chrF, NIST, WER, PER, TER and CDER) as baselines. The collected scores were evaluated in terms of system-level correlation (how well each metric's scores correlate with WMT18 official manual ranking of systems) and in terms of segment-level correlation (how often a metric agrees with humans in judging the quality of a particular sentence relative to alternate outputs). This year, we employ a single kind of manual evaluation: direct assessment (DA).
26
A Study of Translation Edit Rate with Targeted Human Annotation
Matthew Snover, Bonnie J. Dorr, Richard Schwartz et al. · 2006 · 2.4K citations
Statistical Significance Tests for Machine Translation Evaluation.
Philipp Koehn · 2004 · 1.5K citations
chrF: character n-gram F-score for automatic MT evaluation
Maja Popović · 2015 · 970 citations · Full text