Advancing the State of the Art in Open Domain Dialog Systems through the Alexa Prize

TLDR

Building open‑domain conversational systems that enable engaging, topic‑free dialogues is difficult, prompting the launch of the Alexa Prize in 2016 to foster natural, sustained, coherent, and engaging open‑domain conversations. This paper documents the advances made by university teams and the Alexa Prize team toward advancing Conversational AI by tackling key open‑ended challenges such as speech recognition, natural language understanding, commonsense reasoning, dialog management, and evaluation. In the 2018 competition, teams leveraged contextual modeling, knowledge graphs, complex utterance handling, statistical and hierarchical dialog managers, and model‑driven user‑response signals, supported by the CoBot toolkit, topic and dialog‑act detectors, conversation evaluators, and sensitive‑content detection models to build knowledge‑rich, coherent, multi‑turn systems. These efforts raised Alexa user experience metrics—average rating to 3.61, median duration to 2 min 18 s, and average turns to 14.6—up 14%, 92%, and 54% respectively, while speech‑recognition WER fell 55% and entity error 34%, and social‑bot quality improved markedly in 2018 thanks to the CoBot toolkit.

Abstract

Building open domain conversational systems that allow users to have engaging conversations on topics of their choice is a challenging task. Alexa Prize was launched in 2016 to tackle the problem of achieving natural, sustained, coherent and engaging open-domain dialogs. In the second iteration of the competition in 2018, university teams advanced the state of the art by using context in dialog models, leveraging knowledge graphs for language understanding, handling complex utterances, building statistical and hierarchical dialog managers, and leveraging model-driven signals from user responses. The 2018 competition also included the provision of a suite of tools and models to the competitors including the CoBot (conversational bot) toolkit, topic and dialog act detection models, conversation evaluators, and a sensitive content detection model so that the competing teams could focus on building knowledge-rich, coherent and engaging multi-turn dialog systems. This paper outlines the advances developed by the university teams as well as the Alexa Prize team to achieve the common goal of advancing the science of Conversational AI. We address several key open-ended problems such as conversational speech recognition, open domain natural language understanding, commonsense reasoning, statistical dialog management, and dialog evaluation. These collaborative efforts have driven improved experiences by Alexa users to an average rating of 3.61, the median duration of 2 mins 18 seconds, and average turns to 14.6, increases of 14%, 92%, 54% respectively since the launch of the 2018 competition. For conversational speech recognition, we have improved our relative Word Error Rate by 55% and our relative Entity Error Rate by 34% since the launch of the Alexa Prize. Socialbots improved in quality significantly more rapidly in 2018, in part due to the release of the CoBot toolkit.