2005 · 195 citations · 20 references
ComputingComputer ScienceTranslation StudiesComputational LinguisticsInformation RetrievalNatural Language ProcessingHypothesis TestingRanking TestsStatistical SignificanceTest ValidityAutomatic Mt EvaluationDiscriminatory Power AssessmentEvaluationEvaluation MetricsPhrase-based Machine TranslationStatistical Significance TestingTrec EvaluationType I Error
We investigate some pitfalls regarding the discriminatory power of MT evaluation metrics and the accuracy of statistical sig-nificance tests. In a discriminative rerank-ing experiment for phrase-based SMT we show that the NIST metric is more sensi-tive than BLEU or F-score despite their in-corporation of aspects of fluency or mean-ing adequacy into MT evaluation. In an experimental comparison of two statistical significance tests we show that p-values are estimated more conservatively by ap-proximate randomization than by boot-strap tests, thus increasing the likelihood of type-I error for the latter. We point out a pitfall of randomly assessing signif-icance in multiple pairwise comparisons, and conclude with a recommendation to combine NIST with approximate random-ization, at more stringent rejection levels than is currently standard. 1
20
Kishore Papineni, Salim Roukos, Todd J. Ward et al. · 2001 · 21.5K citations · Full text
SRILM - an extensible language modeling toolkit
Andreas Stolcke · 2002 · 4.4K citations
A Systematic Comparison of Various Statistical Alignment Models
Franz Josef Och, Hermann Ney · Computational Linguistics · 2003 · 3.9K citations · Full text
Statistical phrase-based translation
Philipp Koehn, Franz Josef Och, Daniel Marcu · 2003 · 3.3K citations · Full text