Multiple Temporalities of Language and Body in Interaction: Challenges for Transcribing Multimodality

TLDR

The article discusses classic and contemporary challenges for transcription and analysis of multimodal interaction, including gesture, gaze, body arrangements, larger groups, material environments, mobile settings, silent activities, and animal encounters. It focuses on the principles of multimodal conversation analysis, how they can be operationalized in a transcription system, and the analytical and conceptual consequences of transcription choices. The study examines multimodal practices—including material resource mobilization, upper and lower body movements, haptic contacts, and camera movements—using French and Swiss German video data. Precise transcription uncovers complex multimodal arrangements and shows that fine‑grained, distinct temporalities underpin sequential organization as a core principle of action.

Abstract

The article focuses on the principles of multimodal CA, the way they can be operationalized in a transcription system, and the analytical and conceptual consequences of transcription choices. Elaborating on the foundations of multimodal CA and on the basis of video recordings of French and Swiss German encounters, as well as animal interactions, the article discusses classic and contemporary challenges for transcription and analysis, such as beyond gesture and gaze, body arrangements in interactional spaces, larger groups, material environments, mobile settings, silent activities, and animal encounters. It also highlights the diversity of multimodal practices involved: mobilizing occasioned material resources, movements not only of the upper (head, gesture) but also the lower (feet, legs, posterior) parts of the body, haptic contacts touching objects and coparticipants, and camera movements. The precise transcription of relevant details reveals complex arrangements of multimodal resources and gestalts. Their fine-grained, distinct, multiple temporalities constitute the basis of their sequential order—for sequentiality as a fundamental organizational principle of action. Data are in French and Swiss German.