Testing challenges for NLP-intensive bots

Abstract

The popularity of bots is on the rise, with many bots able to interact with users via a chat or voice interface thanks to the embedding of a Natural Language Processing (NLP) component. Still, companies often express concerns about the quality of such bots, as their malfunctioning could have a severe impact on the company revenue or image. Unfortunately, the field of testing NLP-intensive bots is still in its infancy. This paper aims to characterize the testing properties and techniques (and their adaptation) relevant to this type of bots. We believe this will be helpful as a reference framework to compare and evaluate future bot testing research initiatives.