Obtaining Predictive Results with an Objective Evaluation of Spoken Dialogue Systems: Experiments with the DCR Assessment Paradigm

Abstract

The DCR methodology is a framework that proposes a generic and detailed evaluation of spoken dialog systems. We have already detailed (Antoine et al., 1998) the theoretical bases of this paradigm. In this paper, we present some experimental results on spoken language understanding that show the feasibility and the reliability of the DCR evaluation as well as its ability to provide a detailed diagnosis of the system's behaviour. Finally, we highlight the extension of the DCR methodology to dialogue management. 1. Introduction During the last decade, the development of spoken language technologies has gone along with the achievement of large evaluation programs which concern spoken dialogue systems as well as some of their components (speech recognition, spoken language understanding, dialogue management). Generally speaking, this evaluation is based on the computation of quantitative metrics that intend to offer an objective and reproducible survey of the system's behaviour. For insta...