2019 · 124 citations · 31 references
ComputingComputer ScienceArtificial IntelligenceHuman-computer InteractionNatural Language ProcessingConversational InterfacesConversational User InterfacesDialogue SystemsHuman-machine InterfacesNatural Language GenerationText GenerationChatbot InteractionChatbot SystemsChatbotsDialogue Response GenerationEvaluationEvaluation Of Dialogue SystemsUser SatisfactionUser Satisfaction Measurement
Nowadays chatbots have been widely adopted in many industries to automatically answer users' questions and requests via chat interfaces. While it has become much easier to develop a chatbot system, the system itself is a complex system in nature. It is a challenge to evaluate and compare various chatbot systems in terms of effectiveness, efficiency, goal achievability, and the ability to satisfy users. This paper presents a survey, starting from literature review, chatbot architecture, evaluation methods/criteria, and comparison of evaluation methods. Focused on the three subprocesses in the chatbot architecture: text processing, semantic understanding, and response generation. Moreover, the survey is conducted with classification of chatbot evaluation methods and their analysis according to chatbot types and three main evaluation schemes; content evaluation, user satisfaction, and chat function.
31
Kishore Papineni, Salim Roukos, Todd J. Ward et al. · 2001 · 21.5K citations · Full text
ROUGE: A Package for Automatic Evaluation of Summaries
Chin-Yew Lin · 2004 · 8.3K citations
A New Chatbot for Customer Service on Social Media