Pre-Trained Language Models and Their Applications

TLDR

Pre‑trained language models have revolutionized NLP by shifting from supervised learning to a pre‑training plus fine‑tuning paradigm, sparking extensive research into improving these models. This review surveys key advances in pre‑trained models, introduces a taxonomy of such models, and outlines future research directions. The authors describe the architecture of pre‑trained models, their characteristic methods and frameworks, and analyze their impact, challenges, and downstream applications.

Abstract

Pre-trained language models have achieved striking success in natural language processing (NLP), leading to a paradigm shift from supervised learning to pre-training followed by fine-tuning. The NLP community has witnessed a surge of research interest in improving pre-trained models. This article presents a comprehensive review of representative work and recent progress in the NLP field and introduces the taxonomy of pre-trained models. We first give a brief introduction of pre-trained models, followed by characteristic methods and frameworks. We then introduce and analyze the impact and challenges of pre-trained models and their downstream applications. Finally, we briefly conclude and address future research directions in this field.