Demystifying Prompts in Language Models via Perplexity Estimation

Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith, Luke Zettlemoyer

2023 · 92 citations · 24 references

DOIFull text

Open access

Concepts

TL;DR

Language models can perform many tasks via prompting, but performance varies widely and the reasons remain unclear. The study investigates whether prompt performance is predicted by how familiar the model is with the prompt’s language. The authors analyze prompt variance, propose that familiarity predicts performance, and develop a method that expands a seed prompt set through GPT‑3 paraphrasing and back‑translation. Across many tasks, lower prompt perplexity consistently predicts higher performance, and the expanded prompt set confirms perplexity as a strong success predictor.

Abstract

Language models can be prompted to perform a wide variety of tasks with zero- and few-shot in-context learning. However, performance varies significantly with the choice of prompt, and we do not yet understand why this happens. In this paper, we analyze the factors that contribute to this variance and establish a new empirical hypothesis: the performance of a prompt is predicted by the extent to which the model is familiar with the language it contains. Over a wide range of tasks, we show that the lower the perplexity of the prompt, the better it is able to perform the task, when considering reasonable prompts that are related to it. As part of our analysis, we also devise a method to automatically extend a small seed set of manually written prompts by paraphrasing with GPT3 and backtranslation. This larger set allows us to verify that perplexity is a strong predictor of the success of a prompt and we show that the lowest perplexity prompts are consistently effective.

References

24