2023 · 92 citations · 24 references
ComputingComputer ScienceComputational LinguisticsMachine LearningNatural Language ProcessingLanguage ModelingLanguage ModelsMachine TranslationNatural Language UnderstandingSentence ProcessingFew-shot Language ModelingFew-shot LearningIn-context LearningLanguage Model PromptingLanguage Processing ModelsZero-shot Language Modeling
Language models can perform many tasks via prompting, but performance varies widely and the reasons remain unclear. The study investigates whether prompt performance is predicted by how familiar the model is with the prompt’s language. The authors analyze prompt variance, propose that familiarity predicts performance, and develop a method that expands a seed prompt set through GPT‑3 paraphrasing and back‑translation. Across many tasks, lower prompt perplexity consistently predicts higher performance, and the expanded prompt set confirms perplexity as a strong success predictor.
Language models can be prompted to perform a wide variety of tasks with zero- and few-shot in-context learning. However, performance varies significantly with the choice of prompt, and we do not yet understand why this happens. In this paper, we analyze the factors that contribute to this variance and establish a new empirical hypothesis: the performance of a prompt is predicted by the extent to which the model is familiar with the language it contains. Over a wide range of tasks, we show that the lower the perplexity of the prompt, the better it is able to perform the task, when considering reasonable prompts that are related to it. As part of our analysis, we also devise a method to automatically extend a small seed set of manually written prompts by paraphrasing with GPT3 and backtranslation. This larger set allows us to verify that perplexity is a strong predictor of the success of a prompt and we show that the lowest perplexity prompts are consistently effective.
24
Learning Word Vectors for Sentiment Analysis
Andrew L. Maas, Raymond E. Daly, Peter T. Pham et al. · 2011 · 3.3K citations
DBpedia – A large-scale, multilingual knowledge base extracted from Wikipedia
Jens Lehmann, Robert Isele, Max Jakob et al. · Semantic Web · 2015 · 3.2K citations · Full text
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Xiang Lisa Li, Percy Liang · 2021 · 2.2K citations · Full text