2023 · 16 citations · 32 references
ComputingComputer ScienceComputational LinguisticsMachine LearningNatural Language ProcessingLanguage ModelingLanguage ModelsNamed Entity RecognitionNatural Language UnderstandingPart-of-speech TaggingSequence LabelingChunkingFew-shot LearningIn-context LearningLanguage Model PretrainingLanguage Model PromptingPretrainingPromptingStructured Prediction
Although pretrained language models (PLMs) can be prompted to perform a wide range of language tasks, it remains an open question how much this ability comes from generalizable linguistic understanding versus surface-level lexical patterns. To test this, we present a structured prompting approach for linguistic structured prediction tasks, allowing us to perform zero- and few-shot sequence tagging with autoregressive PLMs. We evaluate this approach on part-of-speech tagging, named entity recognition, and sentence chunking, demonstrating strong few-shot performance in all cases. We also find that while PLMs contain significant prior knowledge of task labels due to task leakage into the pretraining corpus, structured prompting can also retrieve linguistic structure with arbitrary labels. These findings indicate that the in-context learning ability and linguistic knowledge of PLMs generalizes beyond memorization of their training data.
32
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Thomas Wolf, Lysandre Debut, Victor Sanh et al. · arXiv (Cornell University) · 2019 · 3.1K citations · Full text
PaLM: Scaling Language Modeling with Pathways
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin et al. · arXiv (Cornell University) · 2022 · 2.1K citations · Full text
Language Models as Knowledge Bases?
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel et al. · 2019 · 1.8K citations · Full text