Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models

TLDR

Prompting language models with training examples and task descriptions is considered essential for recent few‑shot learning successes. The authors aim to determine whether finetuning language models in a few‑shot setting can reduce the need for prompt engineering. They finetune language models on few‑shot tasks, comparing full‑parameter finetuning with bias‑only updates and evaluating performance with null prompts. Null prompts achieve competitive accuracy, bias‑only finetuning updates only 0.1 % of parameters yet matches or surpasses full finetuning, and overall finetuning is more accurate, robust to prompts, and nearly as efficient as frozen models.

Abstract

Prompting language models (LMs) with training examples and task descriptions has been seen as critical to recent successes in few-shot learning. In this work, we show that finetuning LMs in the few-shot setting can considerably reduce the need for prompt engineering. In fact, one can use null prompts, prompts that contain neither task-specific templates nor training examples, and achieve competitive accuracy to manually-tuned prompts across a wide range of tasks. While finetuning LMs does introduce new parameters for each downstream task, we show that this memory overhead can be substantially reduced: finetuning only the bias terms can achieve comparable or better accuracy than standard finetuning while only updating 0.1% of the parameters. All in all, we recommend finetuning LMs for few-shot learning as it is more accurate, robust to different prompts, and can be made nearly as efficient as using frozen LMs.