Vocabulary Agreement Among Model Summaries And Source Documents 1

Abstract

Analysis of 9000 manually-written summaries of newswire stories provided to participants in four Document Understanding Conferences indicates that no more than 55 % of the vocabulary items they employ occur in the source document. A comparison of all pairs of different summaries of the same document shows further that these agree on only 22% of their vocabulary. It can be argued that these relationships establish a performance ceiling for automated summarization systems which compose summaries by extracting syntactic elements from the source document. 1