Robust Part-Based Hand Gesture Recognition Using Kinect Sensor

TLDR

Depth sensors such as the Kinect have expanded human‑computer interaction, yet hand gesture recognition remains difficult because the hand is small, highly articulated, and prone to segmentation errors. This work develops a robust part‑based hand gesture recognition system that operates with the Kinect sensor. The system introduces Finger‑Earth Mover’s Distance, a metric that matches finger parts rather than the whole hand, improving discrimination of subtle gesture differences. Experiments show the system attains 93.2 % mean accuracy on a challenging 10‑gesture set, processes frames in 0.075 s, and remains robust to articulation, distortion, orientation, scale, and cluttered backgrounds, outperforming prior methods in two real‑world HCI applications.

Abstract

The recently developed depth sensors, e.g., the Kinect sensor, have provided new opportunities for human-computer interaction (HCI). Although great progress has been made by leveraging the Kinect sensor, e.g., in human body tracking, face recognition and human action recognition, robust hand gesture recognition remains an open problem. Compared to the entire human body, the hand is a smaller object with more complex articulations and more easily affected by segmentation errors. It is thus a very challenging problem to recognize hand gestures. This paper focuses on building a robust part-based hand gesture recognition system using Kinect sensor. To handle the noisy hand shapes obtained from the Kinect sensor, we propose a novel distance metric, Finger-Earth Mover's Distance (FEMD), to measure the dissimilarity between hand shapes. As it only matches the finger parts while not the whole hand, it can better distinguish the hand gestures of slight differences. The extensive experiments demonstrate that our hand gesture recognition system is accurate (a 93.2% mean accuracy on a challenging 10-gesture dataset), efficient (average 0.0750 s per frame), robust to hand articulations, distortions and orientation or scale changes, and can work in uncontrolled environments (cluttered backgrounds and lighting conditions). The superiority of our system is further demonstrated in two real-life HCI applications.