2014 · 170 citations · 14 references
ComputingComputer ScienceComputational LinguisticsMachine LearningNatural Language ProcessingConversational User InterfacesDialogue SystemsSpeech RecognitionSpoken CorporaDialog SystemsEvaluation Of Dialogue SystemsLanguage UnderstandingSpeech UnderstandingSpoken Dialogue Systems
Dialog state tracking infers a user’s goal from dialog history, a task hampered by speech recognition and language understanding errors yet vital for spoken dialog systems. This paper reports results from the third Dialog State Tracking Challenge, a community competition using a corpus of annotated human‑computer dialogs and a blind test set. The challenge introduced a new test of generalization to unseen entities—new slots and values not in training data—and attracted 28 submissions from seven research teams. Approximately half of the submissions surpassed a competitive rule‑based baseline, demonstrating the effectiveness of statistical methods while highlighting the problem’s difficulty.
In spoken dialog systems, dialog state tracking refers to the task of correctly inferring the user's goal at a given turn, given all of the dialog history up to that turn. This task is challenging because of speech recognition and language understanding errors, yet good dialog state tracking is crucial to the performance of spoken dialog systems. This paper presents results from the third Dialog State Tracking Challenge, a research community challenge task based on a corpus of annotated logs of human-computer dialogs, with a blind test set evaluation. The main new feature of this challenge is that it studied the ability of trackers to generalize to new entities - i.e. new slots and values not present in the training data. This challenge received 28 entries from 7 research teams. About half the teams substantially exceeded the performance of a competitive rule-based baseline, illustrating not only the merits of statistical methods for dialog state tracking but also the difficulty of the problem.
14
The Second Dialog State Tracking Challenge
Matthew Henderson, Blaise Thomson, J. D. Williams · 2014 · 580 citations · Full text
The Dialog State Tracking Challenge
J. D. Williams, Antoine Raux, Deepak Ramachandran et al. · 2013 · 332 citations