Recurrent Continuous Translation Models

TLDR

The paper proposes Recurrent Continuous Translation Models, probabilistic neural translation systems that use continuous representations and avoid alignments or phrase tables. The models generate translations with a target recurrent language model while conditioning on the source sentence through a convolutional sentence model. Experiments show the models achieve over 43 % lower perplexity than alignment‑based systems, are highly sensitive to source word order and syntax, and match state‑of‑the‑art performance when rescoring n‑best lists.

Abstract

We introduce a class of probabilistic continuous translation models called Recurrent Continuous Translation Models that are purely based on continuous representations for words, phrases and sentences and do not rely on alignments or phrasal translation units. The models have a generation and a conditioning aspect. The generation of the translation is modelled with a target Recurrent Language Model, whereas the conditioning on the source sentence is modelled with a Convolutional Sentence Model. Through various experiments, we show first that our models obtain a perplexity with respect to gold translations that is > 43% lower than that of stateof-the-art alignment-based translation models. Secondly, we show that they are remarkably sensitive to the word order, syntax, and meaning of the source sentence despite lacking alignments. Finally we show that they match a state-of-the-art system when rescoring n-best lists of translations.