2023 · 48 citations · 32 references
ComputingComputer ScienceComputational LinguisticsNatural Language ProcessingText AnalyticsData SummarizationLanguage ModelingText GenerationText RepresentationText SummarizationTopic ModelingBenchmark DatasetsHuman Evaluation Of SummariesLarge Language ModelsSummarizationSummarization DatasetsSummarization EvaluationSummary FaithfulnessSummary Generation Models
Existing text summarization systems have made significant progress in recent years, but typically generate summaries in a single step. The one-shot summarization setting is sometimes inadequate, however, as the generated summary may contain hallucinations or overlook important details related to the reader's interests. In this paper, we address this limitation by proposing SummIt, an iterative text summarization framework based on large language models like ChatGPT. Our framework enables the model to refine the generated summary iteratively through self-evaluation and feedback, closely resembling the iterative process humans undertake when drafting and revising summaries. Furthermore, we explore the potential benefits of integrating knowledge and topic extractors into the framework to enhance summary faithfulness and controllability. We evaluate the performance of our framework on three benchmark summarization datasets through empirical and qualitative analyses. We also conduct a human evaluation to validate the effectiveness of the model's refinements and find a potential issue of over-correction.
32
ROUGE: A Package for Automatic Evaluation of Summaries
Chin-Yew Lin · 2004 · 8.3K citations
A technique for measurement of attitudes
Rensis Likert · Medical Entomology and Zoology · 1932 · 8.2K citations