A formal framework for evaluation of information extraction

Abstract

An important problem in the field of Information Extraction (IE) is the lack of clear guidelines for evaluating the correctness of the output generated by an extraction algorithm. This paper tries to handle this problem by providing a formal framework for IE and its evaluation. We define IE in two di#erent, but frequently used approaches: the "All Occurrences" and the "One Best per Document" settings, and we give a formal approach for evaluating an IE system in both settings. Our approach is based on the observation that most commonly used evaluation measures use the confusion matrix as a basis for their computation. We also shortly discuss the most frequently used evaluation measures.