2017 · 203 citations · 12 references
ComputingComputer ScienceArtificial IntelligenceComputational LinguisticsNatural Language ProcessingAnnotationConversational InterfacesConversational User InterfacesDialogue SystemsNatural Language UnderstandingQuestion AnsweringChatbot SystemsChatbotsConversational AgentsConversational Question AnsweringEvaluation Of Dialogue SystemsNatural Language InterfacesQuestion Answering SystemsQuestion Answering Systems Evaluation
Conversational interfaces, especially chatbots that can be built without programming thanks to NLU services, have attracted significant attention, yet systematic analysis of these services is lacking. The study proposes a method to evaluate the classification performance of NLU services, aiming to help researchers and companies choose the most suitable service. The method relies on two newly created corpora—annotated questions and annotated question–answer pairs—and applies them to benchmark popular NLU services. Using these corpora, the authors evaluated several leading NLU services, revealing their relative classification performance.
Conversational interfaces recently gained a lot of attention. One of the reasons for the current hype is the fact that chatbots (one particularly popular form of conversational interfaces) nowadays can be created without any programming knowledge, thanks to different toolkits and so-called Natural Language Understanding (NLU) services. While these NLU services are already widely used in both, industry and science, so far, they have not been analysed systematically. In this paper, we present a method to evaluate the classification performance of NLU services. Moreover, we present two new corpora, one consisting of annotated questions and one consisting of annotated questions with the corresponding answers. Based on these corpora, we conduct an evaluation of some of the most popular NLU services. Thereby we want to enable both, researchers and companies to make more educated decisions about which service they should use.
12
Cheap and fast---but is it good?
Rion Snow, Brendan O’Connor, Daniel Jurafsky et al. · 2008 · 1.9K citations · Full text
Building Natural-Language Generation Systems
Ehud Reiter · ArXiv.org · 1996 · 1.3K citations · Full text
Chris Callison-Burch · 2009 · 426 citations · Full text
Different measurements metrics to evaluate a chatbot system
Bayan Abu Shawar, Eric Atwell · 2007 · 152 citations · Full text