Phrase-Based & Neural Unsupervised Machine Translation

TLDR

Machine translation systems reach near human‑level performance on some languages, but their effectiveness depends on large parallel corpora, limiting applicability to many language pairs. This work investigates how to learn to translate using only large monolingual corpora in each language. We propose neural and phrase‑based models that leverage careful parameter initialization, the denoising effect of language models, and automatic generation of parallel data by iterative back‑translation. The models outperform existing methods, achieving 28.1 and 25.2 BLEU on WMT’14 English‑French and WMT’16 German‑English without any parallel data, surpassing the state of the art by over 11 BLEU, and delivering superior results on low‑resource languages such as English‑Urdu and English‑Romanian, with publicly available code.

Abstract

Machine translation systems achieve near human-level performance on some languages, yet their effectiveness strongly relies on the availability of large amounts of parallel sentences, which hinders their applicability to the majority of language pairs. This work investigates how to learn to translate when having access to only large monolingual corpora in each language. We propose two model variants, a neural and a phrase-based model. Both versions leverage a careful initialization of the parameters, the denoising effect of language models and automatic generation of parallel data by iterative back-translation. These models are significantly better than methods from the literature, while being simpler and having fewer hyper-parameters. On the widely used WMT’14 English-French and WMT’16 German-English benchmarks, our models respectively obtain 28.1 and 25.2 BLEU points without using a single parallel sentence, outperforming the state of the art by more than 11 BLEU points. On low-resource languages like English-Urdu and English-Romanian, our methods achieve even better results than semi-supervised and supervised approaches leveraging the paucity of available bitexts. Our code for NMT and PBSMT is publicly available.