Indian Sign Language recognition system using SURF with SVM and CNN

TLDR

Hand signs provide a natural communication method, especially for the ~1 % of the Indian population with speech impairments, making an automated Indian Sign Language recognition system highly beneficial. The study proposes a Bag of Visual Words approach to recognize Indian Sign Language alphabets and digits from live video, delivering predictions as text and speech. The system segments hand gestures using skin‑color and background subtraction, extracts SURF features to build histograms, and classifies the signs with SVM and CNN, all accessed through an interactive GUI.

Abstract

Hand signs are an effective form of human-to-human communication that has a number of possible applications. Being a natural means of interaction, they are commonly used for communication purposes by speech impaired people worldwide. In fact, about one percent of the Indian population belongs to this category. This is the key reason why it would have a huge beneficial effect on these individuals to incorporate a framework that would understand Indian Sign Language. In this paper, we present a technique that uses the Bag of Visual Words model (BOVW) to recognize Indian sign language alphabets (A-Z) and digits (0–9) in a live video stream and output the predicted labels in the form of text as well as speech. Segmentation is done based on skin colour as well as background subtraction. SURF (Speeded Up Robust Features) features have been extracted from the images and histograms are generated to map the signs with corresponding labels. The Support Vector Machine (SVM) and Convolutional Neural Networks (CNN) are used for classification. An interactive Graphical User Interface (GUI) is also developed for easy access.