The third Dialog State Tracking Challenge

Matthew Henderson, Blaise Thomson, J. D. Williams

2014 · 170 citations · 14 references

Concepts

TL;DR

Dialog state tracking infers a user’s goal from dialog history, a task hampered by speech recognition and language understanding errors yet vital for spoken dialog systems. This paper reports results from the third Dialog State Tracking Challenge, a community competition using a corpus of annotated human‑computer dialogs and a blind test set. The challenge introduced a new test of generalization to unseen entities—new slots and values not in training data—and attracted 28 submissions from seven research teams. Approximately half of the submissions surpassed a competitive rule‑based baseline, demonstrating the effectiveness of statistical methods while highlighting the problem’s difficulty.

Abstract

In spoken dialog systems, dialog state tracking refers to the task of correctly inferring the user's goal at a given turn, given all of the dialog history up to that turn. This task is challenging because of speech recognition and language understanding errors, yet good dialog state tracking is crucial to the performance of spoken dialog systems. This paper presents results from the third Dialog State Tracking Challenge, a research community challenge task based on a corpus of annotated logs of human-computer dialogs, with a blind test set evaluation. The main new feature of this challenge is that it studied the ability of trackers to generalize to new entities - i.e. new slots and values not present in the training data. This challenge received 28 entries from 7 research teams. About half the teams substantially exceeded the performance of a competitive rule-based baseline, illustrating not only the merits of statistical methods for dialog state tracking but also the difficulty of the problem.

References

14