Estimating Machine Translation Post-Editing Effort with HTER

Abstract

Although Machine Translation (MT) has been attracting more and more attention from the translation industry, the quality of current MT systems still requires humans to post-edit translations to ensure their quality. The time necessary to post-edit bad quality trans-lations can be the same or even longer than that of translating without an MT system. It is well known, however, that the quality of an MT system is generally not homoge-neous across all translated segments. In or-der to make MT more useful to the transla-tion industry, it is therefore crucial to have a mechanism to judge MT quality at the seg-ment level to prevent bad quality translations from being post-edited within the translation workflow. We describe an approach to esti-mate translation post-editing effort at sentence level in terms of Human-targeted Translation Edit Rate (HTER) based on a number of fea-tures reflecting the difficulty of translating the source sentence and discrepancies between the source and translation sentences. HTER is a simple metric and obtaining HTER anno-tated data can be made part of the translation workflow. We show that this approach is more reliable at filtering out bad translations than other simple criteria commonly used in the translation industry, such as sentence length. 1