Transactions of the Association for Computational Linguistics · 2024 · 319 citations · 44 references
ComputingComputer ScienceComparative StylisticsComputational LinguisticsNatural Language ProcessingData SummarizationLanguage ModelingLanguage ModelsText SummarizationAutomatic Summarization EvaluationDocument SummarizationHuman Evaluation Of SummariesInstruction TuningLarge Language ModelsNews SummarizationSummarizationSummarization Evaluation
Abstract Large language models (LLMs) have shown promise for automatic summarization but the reasons behind their successes are poorly understood. By conducting a human evaluation on ten LLMs across different pretraining methods, prompts, and model scales, we make two important observations. First, we find instruction tuning, not model size, is the key to the LLM’s zero-shot summarization capability. Second, existing studies have been limited by low-quality references, leading to underestimates of human performance and lower few-shot and finetuning performance. To better evaluate LLMs, we perform human evaluation over high-quality summaries we collect from freelance writers. Despite major stylistic differences such as the amount of paraphrasing, we find that LLM summaries are judged to be on par with human written summaries.
44
Sequence to Sequence Learning with Neural Networks
Ilya Sutskever, Oriol Vinyals, Quoc V. Le · arXiv (Cornell University) · 2014 · 13.4K citations · Full text
Computing, Computer Science, French +14
Training language models to follow instructions with human feedback
Long Ouyang, Jeff Wu, Xu Jiang et al. · arXiv (Cornell University) · 2022 · 4.3K citations · Full text
On the Opportunities and Risks of Foundation Models
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli et al. · arXiv (Cornell University) · 2021 · 2.2K citations · Full text
PaLM: Scaling Language Modeling with Pathways
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin et al. · arXiv (Cornell University) · 2022 · 2.1K citations · Full text