Multimodal coordination

Abstract

The decisive role of interpersonal coordination in social interactions is well established. As human beings, we are "experts" in decoding and producing social signals, learning since our birth. Though, equipping a machine with the same abilities represents a great challenge to design socially-accepted interfaces. Assessing automatically this coordination requires selecting the most relevant features. In this paper, various audio and visual features are investigated to characterize interactional synchrony in a cooperative task. A hierarchical representation of the similarity between extracted features is proposed. The properties of time-domain (correlation coefficients) and frequency-domain (coherence) synchrony measures are confronted. The presented results support the evidence that synchrony is a multimodal phenomenon. Moreover, it can be assessed both on local (short-length windows of interaction) or global scales (entire interaction).