Rapport with Virtual Agents: What Do Human Social Cues and Personality Explain?

TLDR

Rapport is crucial for relationship building, yet its establishment in human‑agent interaction has only recently been explored. This study examines whether a human’s social cues and personality can automatically predict rapport with virtual agents. Experiments with two emotional virtual agents collected audio‑visual recordings, personality assessments, and self‑reported and observer‑rated rapport, from which turn‑taking patterns and facial expressions were extracted. The number of turn‑taking cues and pauses were the strongest predictors of rapport, mirroring findings from human‑human studies, and extraverted and agreeable participants reported higher rapport, indicating that automated social cue analysis can feasibly evaluate agent interactions.

Abstract

Rapport has been recognized as an important aspect of relationship building. While rapport in the context of human-human interaction has been widely studied, how it can be established and maintained in human-agent interaction has been studied only recently. Our study investigates how social cues and personality of a human interacting with an agent can be used for automatic prediction of rapport in this context. We conduct experiments with two emotional virtual agents. Alongside the audio-visual data, we also collect human personality measures and two measures of rapport: self-reported rapport and rapport judged by observers. The social cues, such as turn-taking patterns and facial expressions are extracted from audio-visual data. Our results show that the most significant cues that infer the rapport judgments are the number of turn-taking cues and pauses. We also find that some of the significant social cues related to rapport are similar to those reported in previous psychology literature. We also confirm previous findings on how human personality plays an important role in perceiving the interaction with agents-people who score high in extraversion and agreeableness report higher rapport with both agents. Finally, the rapport prediction results suggest that automatic analysis of social phenomena in human-agent interaction could be a feasible method for agent evaluation.