ComputingComputer ScienceFeature EngineeringInformation RetrievalPerformance MeasurementComparative Performance AnalysisEvaluation Of Retrieval SystemsPerformance EvaluationPerformance MeasuresQuality MetricsAlgorithm Performance EvaluationEfficiency EvaluationEvaluation MetricsInformation Retrieval EvaluationRetrieval Effectiveness MeasuresRetrieval EvaluationSummary MetricsTrec Evaluation Framework
MUC‑3 evaluation metrics, adapted from information‑retrieval and signal‑processing measures, assess performance on the complex template‑fill task, balancing the loss of detail against the need for comparable summary scores over time and across systems. The paper describes how the template‑fill task is scored and how MUC‑3 metrics are calculated to aid analysis of results and future metric development. The authors detail the scoring procedure for the template‑fill task and the computation of MUC‑3 metrics.
The MUC-3 evaluation metrics are measures of performance for the MUC-3 template fill task. Obtaining summary measures of performance necessitates the loss of information about many details of performance. The utility of summary measures for comparison of performance over time and across systems should outweigh this loss of detail. The template fill task is complex because of the varying nature of the fills for each slot and the interdependencies of the slots. The evaluation metrics used in MUC-3 were adapted from traditional measures in information retrieval and signal procesing and were still evolving to fit the more complex data extraction task of MUC-3 when the evaluation was performed. The scoring of the template fill task and the calculation of the metrics used in MUC-3 will be described here. This description is meant to assist in the analysis of the MUC-3 results and in the further evolution of the evaluation metrics.