2019 · 212 citations · 11 references
ComputingComputer ScienceComputational LinguisticsNatural Language ProcessingSyntaxDeep LearningLanguage ModelingLanguage ModelsMultilingual Natural Language ProcessingMultilingual ProcessingParsingFine-tuningLanguage Model Fine-tuningLanguage Model PretrainingLow-resource Language ProcessingModel CapacityMultilingual EmbeddingsMultilingual ParsingNeural Language Models
We show that constituency parsing benefits from unsupervised pre-training across a variety of languages and a range of pre-training conditions. We first compare the benefits of no pre-training, fastText We also find that pre-training is beneficial across all 11 languages tested; however, large model sizes (more than 100 million parameters) make it computationally expensive to train separate models for each language. To address this shortcoming, we show that joint multilingual pre-training and fine-tuning allows sharing all but a small number of parameters between ten languages in the final model. The 10x reduction in model size compared to fine-tuning one model per language causes only a 3.2% relative error increase in aggregate. We further explore the idea of joint fine-tuning and show that it gives low-resource languages a way to benefit from the larger datasets of other languages. Finally, we demonstrate new state-ofthe-art results for 11 languages, including English (95.8 F1) and Chinese (91.8 F1).
11
Constituency Parsing with a Self-Attentive Encoder
Nikita Kitaev, Dan Klein · 2018 · 490 citations · Full text