arXiv (Cornell University) · 2022 · 252 citations · 0 references
ComputingComputer ScienceEducational TechnologyKnowledge RepresentationMachine LearningNatural Language ProcessingAdaptive LearningLanguage ModelingMeta-learningReinforcement LearningSelf-supervised LearningSituated LearningContext-aware Learning SystemsContext-based LearningIn-context Learning
In-context learning enables large language models to predict from a few examples, and recent work has focused on evaluating and extending this paradigm. The paper surveys progress and challenges in in‑context learning. The authors formally define ICL, review training and demonstration strategies, and outline future research directions. They hope the survey will spur further research into how ICL works and how to improve it.
With the increasing ability of large language models (LLMs), in-context learning (ICL) has become a new paradigm for natural language processing (NLP), where LLMs make predictions only based on contexts augmented with a few examples. It has been a new trend to explore ICL to evaluate and extrapolate the ability of LLMs. In this paper, we aim to survey and summarize the progress and challenges of ICL. We first present a formal definition of ICL and clarify its correlation to related studies. Then, we organize and discuss advanced techniques, including training strategies, demonstration designing strategies, as well as related analysis. Finally, we discuss the challenges of ICL and provide potential directions for further research. We hope that our work can encourage more research on uncovering how ICL works and improving ICL.