PERFORMANCE MEASURES FOR INFORMATION EXTRACTION

Abstract

While precision and recall have served the information extraction community well as two separate measures of system performance, we show that the F-measure, the weighted harmonic mean of precision and recall, exhibits certain undesirable behaviors. To overcome these limitations, we define an error measure, the slot error rate, which combines the different types of error directly, without having to resort to precision and recall as preliminary measures. The slot error rate is analogous to the word error rate that is used for measuring speech recognition performance; it is intended to be a measure of the cost to the user for the system to make the different types of errors. 1. INTRODUCTION Precision (P) and recall (R) have been used regularly to measure the performance of information retrieval and information extraction systems. Precision deals with substitution and insertion errors while recall deals with substitution and deletion errors. Because of the community's desire to have a si...