Beyond remote control: Exploring natural gesture inputs for smart TV systems

Abstract

This paper reports on work that explores natural gesture inputs from end-users for television use in a typical living room setting. First, we derive a set of 19 user-defined freehand gestures for regular TV control tasks. This study helps us to determine the gestures preferred by TV viewers and reduce the risk of developing an unusable, ineffective system. Then, based on this user-defined gesture set, we propose a unified framework to address specific problems in a complex real-world TV viewing environment, including 1) the automatic exclusion of many meaningless daily actions by TV viewers, 2) the capability to recognize both static and dynamic gestures simultaneously, as well as one- and two-handed gestures simultaneously, and 3) the continuous recognition of multiple dynamic gestures in the air (e.g., a channel switching gesture for channel 127). Experimental results show that our approach allows users to interact with TV-based applications more flexibly and effectively, with improved user experience and user satisfaction. Finally, we highlight the implications of our work for the design and development of related freehand gesture applications.