Concepedia

BERTje: A Dutch BERT Model

Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, Malvina Nissim

arXiv (Cornell University) · 2019 · 215 citations · 9 references

Open access

Concepts

ComputingComputer ScienceComputational LinguisticsKnowledge RepresentationNatural Language ProcessingLanguage ModelingLanguage ModelsMultilingual Natural Language ProcessingNamed Entity RecognitionPart-of-speech TaggingSemantic Role LabelingLanguage Model PretrainingLarge Language ModelsNeural Language ModelsSemantic LabelingSemantic Role Labeling EvaluationTransformer Language ModelsTransformer-based Named Entity RecognitionTransformer-based Tagging

TL;DR

BERT, a transformer‑based pre‑trained language model, has advanced state‑of‑the‑art performance on many NLP tasks, yet multilingual BERT’s Dutch coverage is limited to Wikipedia text. The authors aim to develop and evaluate a monolingual Dutch BERT model, BERTje, using the same architecture and parameters as the original BERT. They trained BERTje on a large, diverse dataset of 2.4 billion tokens, employing the same architecture and hyperparameters as the original BERT. BERTje consistently outperforms the equally‑sized multilingual BERT model on downstream NLP tasks such as part‑of‑speech tagging, named‑entity recognition, semantic role labeling, and sentiment analysis, and the pre‑trained model is publicly available on GitHub.

Abstract

The transformer-based pre-trained language model BERT has helped to improve state-of-the-art performance on many natural language processing (NLP) tasks. Using the same architecture and parameters, we developed and evaluated a monolingual Dutch BERT model called BERTje. Compared to the multilingual BERT model, which includes Dutch but is only based on Wikipedia text, BERTje is based on a large and diverse dataset of 2.4 billion tokens. BERTje consistently outperforms the equally-sized multilingual BERT model on downstream NLP tasks (part-of-speech tagging, named-entity recognition, semantic role labeling, and sentiment analysis). Our pre-trained Dutch BERT model is made available at https://github.com/wietsedv/bertje.