ERNIE 2.0: A Continual Pre-Training Framework for Language Understanding

TLDR

Pre‑trained language models have achieved state‑of‑the‑art performance, yet most pre‑training focuses on simple co‑occurrence tasks while overlooking valuable lexical, syntactic, and semantic signals in corpora. This work introduces ERNIE 2.0, a continual pre‑training framework that incrementally builds and learns from a sequence of lexical, syntactic, and semantic tasks. ERNIE 2.0 constructs multiple tasks and trains a model to capture lexical, syntactic, and semantic information through continual multi‑task learning. On 16 benchmark tasks—including English GLUE and Chinese analogues—ERNIE 2.0 surpasses BERT and XLNet, and its code and models are publicly available on GitHub.

Abstract

Recently pre-trained models have achieved state-of-the-art results in various language understanding tasks. Current pre-training procedures usually focus on training the model with several simple tasks to grasp the co-occurrence of words or sentences. However, besides co-occurring information, there exists other valuable lexical, syntactic and semantic information in training corpora, such as named entities, semantic closeness and discourse relations. In order to extract the lexical, syntactic and semantic information from training corpora, we propose a continual pre-training framework named ERNIE 2.0 which incrementally builds pre-training tasks and then learn pre-trained models on these constructed tasks via continual multi-task learning. Based on this framework, we construct several tasks and train the ERNIE 2.0 model to capture lexical, syntactic and semantic aspects of information in the training data. Experimental results demonstrate that ERNIE 2.0 model outperforms BERT and XLNet on 16 tasks including English tasks on GLUE benchmarks and several similar tasks in Chinese. The source codes and pre-trained models have been released at https://github.com/PaddlePaddle/ERNIE.