2007 · 385 citations · 42 references
HumanitiesFrenchSpanishCorpus-based Translation StudiesTranslation TheoryTranslator TrainingMachine TranslationTranslation Error AnalysisTranslation Quality AssessmentAnnotator AgreementAutomatic Mt EvaluationEvaluation MetricsHuman Evaluation Of Machine TranslationInter-annotator AgreementInter-observer AgreementMachine Translation Quality EvaluationTranslation Quality Metrics
This paper evaluates the translation quality of machine translation systems for 8 language pairs: translating French, German, Spanish, and Czech to English and back. We carried out an extensive human evaluation which allowed us not only to rank the different MT systems, but also to perform higher-level analysis of the evaluation process. We measured timing and intra- and inter-annotator agreement for three types of subjective evaluation. We measured the correlation of automatic evaluation metrics with human judgments. This meta-evaluation reveals surprising facts about the most commonly used methodologies.
42
The Measurement of Observer Agreement for Categorical Data
J. Richard Landis, Gary G. Koch · Biometrics · 1977 · 78.9K citations · Full text
Kishore Papineni, Salim Roukos, Todd J. Ward et al. · 2001 · 21.5K citations · Full text
A Systematic Comparison of Various Statistical Alignment Models
Franz Josef Och, Hermann Ney · Computational Linguistics · 2003 · 3.9K citations · Full text
Statistical phrase-based translation
Philipp Koehn, Franz Josef Och, Daniel Marcu · 2003 · 3.3K citations · Full text