User-defined gestures for surface computing

TLDR

Surface computing prototypes have largely relied on designer-created gestures, which may not reflect natural user behavior. The study proposes a method that elicits user‑defined tabletop gestures by first demonstrating a gesture’s effect and then asking users to perform its cause. Researchers collected 1,080 gestures from 20 participants, analyzed them alongside think‑aloud data for 27 one‑hand and two‑hand commands. Users rarely consider finger count, prefer one‑hand gestures, and are influenced by desktop idioms; some commands show low agreement, indicating a need for on‑screen widgets, and the authors provide a gesture set, agreement scores, taxonomy, and design implications.

Abstract

Many surface computing prototypes have employed gestures created by system designers. Although such gestures are appropriate for early investigations, they are not necessarily reflective of user behavior. We present an approach to designing tabletop gestures that relies on eliciting gestures from non-technical users by first portraying the effect of a gesture, and then asking users to perform its cause. In all, 1080 gestures from 20 participants were logged, analyzed, and paired with think-aloud data for 27 commands performed with 1 and 2 hands. Our findings indicate that users rarely care about the number of fingers they employ, that one hand is preferred to two, that desktop idioms strongly influence users' mental models, and that some commands elicit little gestural agreement, suggesting the need for on-screen widgets. We also present a complete user-defined gesture set, quantitative agreement scores, implications for surface technology, and a taxonomy of surface gestures. Our results will help designers create better gesture sets informed by user behavior.