2022 · 18 citations · 28 references
ComputingComputer ScienceComputational LinguisticsCorpus-based Translation StudiesNatural Language ProcessingCross-linguistic TransferLanguage ModelingMachine TranslationMultilingual Natural Language ProcessingMultilingual ProcessingNamed Entity RecognitionContextual Word RepresentationCross-lingual Language ModelingCross-lingual Representation LearningCross-lingual TransferCross-lingual Transfer LearningLow-resource Language ProcessingLow-resource Named Entity RecognitionMonolingual Corpora
Pre-trained multilingual language models are the foundation of many NLP approaches, including cross-lingual transfer solutions. However, languages with small available monolingual corpora are often not well-supported by these models leading to poor performance. We propose an unsupervised approach to improve the cross-lingual representations of low-resource languages by bootstrapping word translation pairs from monolingual corpora and using them to improve language alignment in pre-trained language models. We perform experiments on nine languages, using contextual word retrieval and zero-shot named entity recognition to measure both intrinsic cross-lingual word representation quality and downstream task performance, showing improvements on both tasks. Our results show that it is possible to improve pre-trained multilingual language models by relying only on non-parallel resources.
28
Transformers: State-of-the-Art Natural Language Processing
Thomas Wolf, Lysandre Debut, Victor Sanh et al. · 2020 · 8K citations · Full text
Cross-lingual Language Model Pretraining
Guillaume Lample, Alexis Conneau · arXiv (Cornell University) · 2019 · 1.6K citations · Full text