A real-time system to recognize static gestures of Brazilian sign language (libras) alphabet using Kinect

Abstract

In this paper we present a system - called Gesture User Interface (GestureUI) - to recognize static gestures of the Brazilian Sign Language (Libras) in real-time out of a continuous video stream using the depth information captured by a Kinect controller. We focus on handling small sets of gestures (A, E, I, O, U) and (B, C, F, L, V), processing them in two steps: Segmentation and Classification. For the Segmentation we propose a Virtual Wall and Libras-specific heuristics to improve the hand tracking. For the classification we use a Multi-Layer Perceptron trained by the system and present an arm cutting algorithm that improved the recognition rate from 67.4% and 75.4% to 100% for both gesture sets. Finally, we evaluated the processing performance of the overall system and proof that it is able to process frames at 62.5Hz with an Intel i7 processor, which is more than twice as fast as the Kinect frame capturing rate.