Egocentric Gesture Recognition for Head-Mounted AR devices

Abstract

Natural interaction with virtual objects in AR/VR environments\nmakes for a smooth user experience. Gestures are a natural extension\nfrom real world to augmented space to achieve these interactions.\nFinding discriminating spatio-temporal features relevant to gestures\nand hands in ego-view is the primary challenge for recognising\negocentric gestures. In this work we propose a data driven end-toend deep learning approach to address the problem of egocentric\ngesture recognition, which combines an ego-hand encoder network\nto find ego-hand features, and a recurrent neural network to discern\ntemporally discriminating features. Since deep learning networks\nare data intensive, we propose a novel data augmentation technique\nusing green screen capture to alleviate the problem of ground truth\nannotation. In addition we publish a dataset of 10 gestures performed\nin a natural fashion in front of a green screen for training and the\nsame 10 gestures performed in different natural scenes without green\nscreen for validation. We also present the results of our network’s\nperformance in comparison to the state-of-the-art using the AirGest\ndataset.