Realtime and Robust Hand Tracking from Depth

TLDR

We present a realtime hand tracking system using a depth sensor. The system uses several novel techniques, modeling the hand with spheres and a fast cost function, a hybrid gradient‑based and stochastic optimization for fast convergence, and new finger detection and initialization methods to enhance robustness. The system tracks a fully articulated hand in real time at 25 FPS on a desktop without a GPU, achieving sub‑10 mm error, and is the first to combine such robustness, accuracy, and speed, as demonstrated on challenging real data.

Abstract

We present a realtime hand tracking system using a depth sensor. It tracks a fully articulated hand under large viewpoints in realtime (25 FPS on a desktop without using a GPU) and with high accuracy (error below 10 mm). To our knowledge, it is the first system that achieves such robustness, accuracy, and speed simultaneously, as verified on challenging real data. Our system is made of several novel techniques. We model a hand simply using a number of spheres and define a fast cost function. Those are critical for realtime performance. We propose a hybrid method that combines gradient based and stochastic optimization methods to achieve fast convergence and good accuracy. We present new finger detection and hand initialization methods that greatly enhance the robustness of tracking.