Smartwatch-based Early Gesture Detection 8 Trajectory Tracking for Interactive Gesture-Driven Applications

Abstract

The paper explores the possibility of using wrist-worn devices (specifically, a smartwatch) to accurately track the hand movement and gestures for a new class of immersive, interactive gesture-driven applications. These interactive applications need two special features: (a) the ability to identify gestures from a continuous stream of sensor data early--i.e., even before the gesture is complete, and (b) the ability to precisely track the hand's trajectory, even though the underlying inertial sensor data is noisy. We develop a new approach that tackles these requirements by first building a HMM-based gesture recognition framework that does not need an explicit segmentation step, and then using a per-gesture trajectory tracking solution that tracks the hand movement only during these predefined gestures. Using an elaborate setup that allows us to realistically study the table-tennis related hand movements of users, we show that our approach works: (a) it can achieve 95% stroke recognition accuracy. Within 50% of gesture, it can achieve a recall value of 92% for 10 novice users and 93% for 15 experienced users from a continuous sensor stream; (b) it can track hand movement during such strokeplay with a median accuracy of 6.2 cm.