Introduction: Multimodal interaction

TLDR

Human social interaction relies on the intertwined cooperation of different modalities, and recent research has begun to document how talk, gesture, gaze, and material surroundings combine to form coherent courses of action. This volume aims to develop and present evidence for the multimodal nature of social interaction. The authors employ diverse analytic traditions and review literature on communicative functions and interactive organization of modalities to illuminate multimodal social interaction. Recent studies show that modalities jointly elaborate semantic content and constitute coherent courses of action.

Abstract

That human social interaction involves the intertwined cooperation of different modalities is uncontroversial. Researchers in several allied fields have, however, only recently begun to document the precise ways in which talk, gesture, gaze, and aspects of the material surround are brought together to form coherent courses of action. The papers in this volume are attempts to develop this line of inquiry. Although the authors draw on a range of analytic, theoretical, and methodological traditions (conversation analysis, ethnography, distributed cognition, and workplace studies), all are concerned to explore and illuminate the inherently multimodal character of social interaction. Recent studies, including those collected in this volume, suggest that different modalities work together not only to elaborate the semantic content of talk but also to constitute coherent courses of action. In this introduction we present evidence for this position. We begin by reviewing some select literature focusing primarily on communicative functions and interactive organizations of specific modalities before turning to consider the integration of distinct modalities in interaction.