K-BERT: Enabling Language Representation with Knowledge Graph

TLDR

BERT learns general language representations but lacks domain knowledge, and while experts use relevant knowledge to interpret domain text, excessive knowledge injection can introduce knowledge noise that distorts meaning. The authors propose K‑BERT, a knowledge‑enabled language representation model that injects knowledge graph triples into sentences to give machines domain knowledge. K‑BERT injects KG triples into sentences, uses soft‑position and visible matrices to mitigate knowledge noise, and leverages pre‑trained BERT parameters to enable easy domain knowledge integration without additional pre‑training. K‑BERT achieves promising results across twelve NLP tasks and significantly outperforms BERT on domain‑specific tasks such as finance, law, and medicine.

Abstract

Pre-trained language representation models, such as BERT, capture a general language representation from large-scale corpora, but lack domain-specific knowledge. When reading a domain text, experts make inferences with relevant knowledge. For machines to achieve this capability, we propose a knowledge-enabled language representation model (K-BERT) with knowledge graphs (KGs), in which triples are injected into the sentences as domain knowledge. However, too much knowledge incorporation may divert the sentence from its correct meaning, which is called knowledge noise (KN) issue. To overcome KN, K-BERT introduces soft-position and visible matrix to limit the impact of knowledge. K-BERT can easily inject domain knowledge into the models by being equipped with a KG without pre-training by itself because it is capable of loading model parameters from the pre-trained BERT. Our investigation reveals promising results in twelve NLP tasks. Especially in domain-specific tasks (including finance, law, and medicine), K-BERT significantly outperforms BERT, which demonstrates that K-BERT is an excellent choice for solving the knowledge-driven problems that require experts.