American sign language translation using edge detection and cross correlation

Abstract

Currently, the American Sign Language (ASL), expressed through the use of body gestures (hand, face, torso) and perceived through the eyes, is the standard language of communication used by the Deaf community. Our main objective is to implement an automated translation system that is capable of translating ASL to English text using common computing environments such as a computer and a generic webcam. In this paper, we present a real-time hand gesture recognition system using a combination of image processing modalities. A prototype graphical user interface application for ASL sign capture, processing, collection and analysis is presented. The approach consists of a gesture extraction phase followed by a gesture recognition phase. An image gesture database is collected through the application and used as training information to be used in the gesture recognition stage. We provide two different translation paradigms: 1) English characters (alphabet) and 2) complete words or phrases. In the method to recognize individual characters, the hand gesture image is processed by combining image segmentation and edge detection to extract morphological information and then processed by the gesture detection stage that recognizes the corresponding alphabet letter. The translation of words and phrases consists of splitting a video sequence into frames and pre-process them in a feature (frame) selection stage. In this feature selection stage, a subset of frames that can represent a particular word or phrase are selected. The collection of frames representing a word or a phrase are then processed using the multi-modality technique used for processing individual characters. Finally, the gesture recognition stage is applied to both approaches using a cross-correlation coefficient based scheme to detect the expression.