Gesture recognition with a Wii controller

TLDR

User interaction is shifting from mouse and pen to more physical, tangible methods, with emerging technologies enabling intuitive human‑computer interaction. The paper develops and evaluates a sensor‑based gesture recognition system to enable application interaction, laying groundwork for multimodal media browsing. Using the Wii controller’s acceleration sensor and a hidden Markov model, the system trains and recognizes arbitrary gestures, allowing users to interact with applications such as photo browsing on a home TV. Evaluation shows the system can recognize gestures with only a few training samples, and the implementation is available to other researchers.

Abstract

In many applications today user interaction is moving away from mouse and pens and is becoming pervasive and much more physical and tangible. New emerging interaction technologies allow developing and experimenting with new interaction methods on the long way to providing intuitive human computer interaction. In this paper, we aim at recognizing gestures to interact with an application and present the design and evaluation of our sensor-based gesture recognition. As input device we employ the Wii-controller (Wiimote) which recently gained much attention world wide. We use the Wiimote's acceleration sensor independent of the gaming console for gesture recognition. The system allows the training of arbitrary gestures by users which can then be recalled for interacting with systems like photo browsing on a home TV. The developed library exploits Wii-sensor data and employs a hidden Markov model for training and recognizing user-chosen gestures. Our evaluation shows that we can already recognize gestures with a small number of training samples. In addition to the gesture recognition we also present our experiences with the Wii-controller and the implementation of the gesture recognition. The system forms the basis for our ongoing work on multimodal intuitive media browsing and are available to other researchers in the field.