Reference Bias in Monolingual Machine Translation Evaluation

Abstract

In the translation industry, human translations are assessed by comparison with the source texts. In the Machine Translation (MT) research community, however, it is a common practice to perform quality assessment using a reference translation instead of the source text. In this paper we show that this practice has a serious issue -annotators are strongly biased by the reference translation provided, and this can have a negative impact on the assessment of MT quality.