How Personal Experience and Technical Knowledge Affect Using Conversational Agents

Abstract

Conversational agents (CA) use dialogues to interact with users so as to offer an experience of naturalistic interaction. However, due to the low transparency and poor explanability of mechanism inside CA, individual's understanding of CA's capabilities may affect how the individual interacts with CA and the sustainability of CA use. To examine how users' understanding affect perceptions and experiences of using CA, we conducted a laboratory study asking 41 participants performed a set of tasks using Apple Siri. We independently manipulated two factors: (1) personal experience of using CA, and (2) technical knowledge about CA's system model. We conducted mixed-method analyses of post-task usability measures and interviews, and confirmed that use experience and technical knowledge affects perceived usability and mental models differently.