Parameter-efficient fine-tuning of large-scale pre-trained language models

TLDR

Large pre‑trained language models improve performance as they scale, but fine‑tuning and storing all parameters becomes prohibitively expensive, motivating research into parameter‑efficient adaptation that optimizes only a small fraction of weights. This work introduces delta‑tuning, a unified framework that formalizes the problem of optimizing a small subset of parameters while keeping the rest fixed, and proposes a categorization criterion for existing methods. The authors analyze theoretical principles from optimization and optimal control, and conduct a comprehensive empirical study across more than 100 NLP tasks to evaluate delta‑tuning methods. The study confirms that delta‑tuning offers both theoretical soundness and practical effectiveness for adapting large‑scale language models with minimal parameter updates.

Abstract

Abstract With the prevalence of pre-trained language models (PLMs) and the pre-training–fine-tuning paradigm, it has been continuously shown that larger models tend to yield better performance. However, as PLMs scale up, fine-tuning and storing all the parameters is prohibitively costly and eventually becomes practically infeasible. This necessitates a new branch of research focusing on the parameter-efficient adaptation of PLMs, which optimizes a small portion of the model parameters while keeping the rest fixed, drastically cutting down computation and storage costs. In general, it demonstrates that large-scale models could be effectively stimulated by the optimization of a few parameters. Despite the various designs, here we discuss and analyse the approaches under a more consistent and accessible term ‘delta-tuning’, where ‘delta’ a mathematical notation often used to denote changes, is borrowed to refer to the portion of parameters that are ‘changed’ during training. We formally describe the problem and propose a unified categorization criterion for existing delta-tuning methods to explore their correlations and differences. We also discuss the theoretical principles underlying the effectiveness of delta-tuning and interpret them from the perspectives of optimization and optimal control. Furthermore, we provide a holistic empirical study on over 100 natural language processing tasks and investigate various aspects of delta-tuning. With comprehensive study and analysis, our research demonstrates the theoretical and practical properties of delta-tuning in the adaptation of PLMs.