Developing a Clinical Language Model for Swedish: Continued Pretraining of Generic BERT with In-Domain Data

Abstract

The use of pretrained language models, finetuned to perform a specific downstream task, has become widespread in NLP. Using a generic language model in specialized domains may, however, be sub-optimal due to differences in language use and vocabulary. In this paper, it is investigated whether an existing, generic language model for Swedish can be improved for the clinical domain through continued pretraining with clinical text.