Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation

Concepts

ComputingTranslation StudiesKnowledge TransferTranslation TheoryTranslator TrainingLanguage TransferMachine TranslationTransfer LearningCross-lingual TransferCross-lingual Transfer LearningLow-resource Language ProcessingLow-resource Machine TranslationNeural Machine TranslationWord-based Machine Translation

TL;DR

The transfer method builds on Zoph et al.’s approach by exploiting source‑vocabulary overlap rather than ignoring it. The authors propose a simple method to improve neural translation of a low‑resource language pair by leveraging parallel data from a related, also low‑resource, pair. They first apply Byte Pair Encoding to increase vocabulary overlap, then train a model on the first pair, transfer its parameters—including source‑word embeddings—to a second model, and continue training on the second pair. Experiments show that transfer learning alone only slightly benefits word‑based translation, but when combined with a strong BPE baseline it yields up to a 4.3‑point BLEU improvement.

Abstract

We present a simple method to improve neural translation of a low-resource language pair using parallel data from a related, also low-resource, language pair. The method is based on the transfer method of Zoph et al., but whereas their method ignores any source vocabulary overlap, ours exploits it. First, we split words using Byte Pair Encoding (BPE) to increase vocabulary overlap. Then, we train a model on the first language pair and transfer its parameters, including its source word embeddings, to another model and continue training on the second language pair. Our experiments show that transfer learning helps word-based translation only slightly, but when used on top of a much stronger BPE baseline, it yields larger improvements of up to 4.3 BLEU.