1998 Broadcast News Benchmark Test Results: English and Non-English Word Error Rate Performance Measures

Abstract

This paper documents the use of Broadcast News test materials in DARPA-sponsored Automatic Speech Recognition (ASR) Benchmark Tests conducted late in 1998. As in last year's tests [1], statistical selection procedures were used in selecting test materials. Two test epochs were used, each yielding (nominally) one and one-half hours of test material. One of the test sets was drawn from the same test epoch as was used for last year's tests, and the other was drawn from a more recent period. Results are reported for two types of systems: one (the "Hub", or "baseline" systems) for which there were no limits on computational resources, and another (the "less than 10X realtime spoke" systems) for systems that ran in less than 10 times real-time. The lowest word error rate reported this year for the "Hub" systems was 13.5%, contrasting with last year's lowest word error rate of 16.2%. For the "less than 10X real-time spoke" systems, the lowest reported word error rate was 16.1%. Results are a...