A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies

TLDR

Machine‑learning dialogue managers rely on automatic user simulation to explore vast strategy spaces, prompting research into statistical user models such as Markov chains and Bayesian networks and raising challenges like concept drift and evaluation. This survey outlines the dialogue manager’s role, introduces reinforcement‑learning for strategy design, and reviews user‑modeling techniques used in simulation‑based learning. It also surveys recent advances in user‑model evaluation and highlights open research problems in simulation‑based reinforcement learning from a user‑modeling viewpoint.

Abstract

Within the broad field of spoken dialogue systems, the application of machine-learning approaches to dialogue management strategy design is a rapidly growing research area. The main motivation is the hope of building systems that learn through trial-and-error interaction what constitutes a good dialogue strategy. Training of such systems could in theory be done using human users or using corpora of human–computer dialogue, but in practice the typically vast space of possible dialogue states and strategies cannot be explored without the use of automatic user simulation tools. This requirement for training statistical dialogue models has created an interesting new application area for predictive statistical user modelling and a variety of different techniques for simulating user behaviour have been presented in the literature ranging from simple Markov models to Bayesian networks. The development of reliable user simulation tools is critical to further progress on automatic dialogue management design but it holds many challenges, some of which have been encountered in other areas of current research on statistical user modelling, such as the problem of ‘concept drift’, the problem of combining content-based and collaboration-based modelling techniques, and user model evaluation. The latter topic is of particular interest, because simulation-based learning is currently one of the few applications of statistical user modelling that employs both direct ‘accuracy-based’ and indirect ‘utility-based’ evaluation techniques. In this paper, we briefly summarize the role of the dialogue manager in a spoken dialogue system, give a short introduction to reinforcement-learning of dialogue management strategies and review the literature on user modelling for simulation-based strategy learning. We further describe recent work on user model evaluation and discuss some of the current research issues in simulation-based learning from a user modelling perspective.