A stochastic model of human-machine interaction for learning dialog strategies

TLDR

Dialog design can be formalized as an optimization problem over a sequential decision process, and data‑driven reinforcement learning methods within the MDP framework are available to find optimal strategies. The study proposes a quantitative model and a combined supervised–reinforcement learning approach to learn dialog strategies for dialog systems. The method maps dialog systems to MDPs, estimates user behavior with supervised learning, then applies reinforcement learning to derive optimal strategies, and is evaluated on an air‑travel information system task. Experiments demonstrate that a simple criterion, state‑space representation, and simulated user model enable automatic learning of complex dialog behavior comparable to heuristic designs.

Abstract

We propose a quantitative model for dialog systems that can be used for learning the dialog strategy. We claim that the problem of dialog design can be formalized as an optimization problem with an objective function reflecting different dialog dimensions relevant for a given application. We also show that any dialog system can be formally described as a sequential decision process in terms of its state space, action set, and strategy. With additional assumptions about the state transition probabilities and cost assignment, a dialog system can be mapped to a stochastic model known as Markov decision process (MDP). A variety of data driven algorithms for finding the optimal strategy (i.e., the one that optimizes the criterion) is available within the MDP framework, based on reinforcement learning. For an effective use of the available training data we propose a combination of supervised and reinforcement learning: the supervised learning is used to estimate a model of the user, i.e., the MDP parameters that quantify the user's behavior. Then a reinforcement learning algorithm is used to estimate the optimal strategy while the system interacts with the simulated user. This approach is tested for learning the strategy in an air travel information system (ATIS) task. The experimental results we present in this paper show that it is indeed possible to find a simple criterion, a state space representation, and a simulated user parameterization in order to automatically learn a relatively complex dialog behavior, similar to one that was heuristically designed by several research groups.