Gestures Over Video Streams to Support Remote Collaboration on Physical Tasks

TLDR

Remote collaboration on physical tasks relies on video systems, yet effective conversational grounding requires both pointing and representational gestures to refer to objects and actions. This study investigates how to instantiate pointing and representational gestures within remote collaboration systems. Two studies compared surrogate gesture tools—a cursor‑based pointing device and a pen‑based drawing tool—to video‑only conditions, measuring performance side‑by‑side. Results show that surrogate gesture tools are effective only when they support both pointing and representational gestures, and that an automatic erasure function improves collaboration; the study discusses implications and future research directions.

Abstract

Abstract This article considers tools to support remote gesture in video systems being used to complete collaborative physical tasks-tasks in which two or more individuals work together manipulating three-dimensional objects in the real world. We first discuss the process of conversational grounding during collaborative physical tasks, particularly the role of two types of gestures in the grounding process: pointing gestures, which are used to refer to task objects and locations, and representational gestures, which are used to represent the form of task objects and the nature of actions to be used with those objects. We then consider ways in which both pointing and representational gestures can be instantiated in systems for remote collaboration on physical tasks. We present the results of two studies that use a "surrogate" approach to remote gesture, in which images are intended to express the meaning of gestures through visible embodiments, rather than direct views of the hands. In Study 1, we compare performance with a cursor-based pointing device that allows remote partners to point to objects in a video feed of the work area to performance side-by-side or with the video system alone. In Study 2, we compare performance with two variations of a pen-based drawing tool that allows for both pointing and representational gestures to performance with video alone. The results suggest that simple surrogate gesture tools can be used to convey gestures from remote sites, but that the tools need to be able to convey representational as well as pointing gestures to be effective. The results further suggest that an automatic erasure function, in which drawings disappear a few seconds after they were created, is more beneficial for collaboration than tools requiring manual erasure. We conclude with a discussion of the theoretical and practical implications of the results, as well as several areas for future research.