A new quantitative quality measure for machine translation systems

Abstract

In this paper, an objective quantitative quality measure is proposed to evaluate the performance of machine translation systems. The proposed method is to compare the raw translation output of an MT system with the final revised version for the customers, and then compute the editing efforts required to convert the raw translation to the final version. In contrast to the other proposals, the evaluation process can be done quickly and automatically. Hence, it can provide a quick response on any system change. A system designer can thus quickly find the advantages or faults of a particular performance dynamically. Application of such a measure to improve the system performance on-line on a parameterized and feedback-controlled system will be demonstrated. Furthermore, because the revised version is used directly as a reference, the performance measure can reflect the real quality gap between the system performance and customer expectation. A system designer can thus concentrate on practically important topics rather than on theoretically interesting issues.