Real-time input of 3D pose and gestures of a user's hand and its applications for HCI

Abstract

Introduces a method for tracking a user's hand in 3D and recognizing the hand's gesture in real time without the use of any invasive devices attached to the hand. Our method uses multiple cameras for determining the position and orientation of a user's hand moving freely in a 3D space. In addition, the method identifies pre-determined gestures in a fast and robust manner by using a neural network which has been properly trained beforehand. This paper also describes results of user study of our proposed method and several types of applications, including 3D object handling for a desktop system and a 3D walkthrough for a large immersive display system.