A Hand Gesture Recognition Framework and Wearable Gesture-Based Interaction Prototype for Mobile Devices

TLDR

The study proposes an algorithmic framework to process acceleration and surface electromyographic signals for gesture recognition. The framework employs a novel segmentation scheme, score‑based sensor fusion, two new features, a Bayes linear classifier, and an improved dynamic time‑warping algorithm, and is implemented in a wearable prototype with a three‑axis accelerometer and four SEMG sensors that enables real‑time gesture‑based interaction on a mobile phone. The prototype responded to gestures within 300 ms, achieving 95.0 % accuracy in user‑dependent tests and 89.6 % in user‑independent tests, and users reported positive experience.

Abstract

An algorithmic framework is proposed to process acceleration and surface electromyographic (SEMG) signals for gesture recognition. It includes a novel segmentation scheme, a score-based sensor fusion scheme, and two new features. A Bayes linear classifier and an improved dynamic time-warping algorithm are utilized in the framework. In addition, a prototype system, including a wearable gesture sensing device (embedded with a three-axis accelerometer and four SEMG sensors) and an application program with the proposed algorithmic framework for a mobile phone, is developed to realize gesture-based real-time interaction. With the device worn on the forearm, the user is able to manipulate a mobile phone using 19 predefined gestures or even personalized ones. Results suggest that the developed prototype responded to each gesture instruction within 300 ms on the mobile phone, with the average accuracy of 95.0% in user-dependent testing and 89.6% in user-independent testing. Such performance during the interaction testing, along with positive user experience questionnaire feedback, demonstrates the utility of the framework.