Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language\n Model

Abstract

Recent breakthroughs of pretrained language models have shown the\neffectiveness of self-supervised learning for a wide range of natural language\nprocessing (NLP) tasks. In addition to standard syntactic and semantic NLP\ntasks, pretrained models achieve strong improvements on tasks that involve\nreal-world knowledge, suggesting that large-scale language modeling could be an\nimplicit method to capture knowledge. In this work, we further investigate the\nextent to which pretrained models such as BERT capture knowledge using a\nzero-shot fact completion task. Moreover, we propose a simple yet effective\nweakly supervised pretraining objective, which explicitly forces the model to\nincorporate knowledge about real-world entities. Models trained with our new\nobjective yield significant improvements on the fact completion task. When\napplied to downstream tasks, our model consistently outperforms BERT on four\nentity-related question answering datasets (i.e., WebQuestions, TriviaQA,\nSearchQA and Quasar-T) with an average 2.7 F1 improvements and a standard\nfine-grained entity typing dataset (i.e., FIGER) with 5.7 accuracy gains.\n