American sign language recognition with the kinect

TLDR

The study explores using the Kinect depth‑mapping camera to recognize and verify American Sign Language for educational games for deaf children, aiming to enhance interactivity, comfort, robustness, sustainability, cost, and deployment ease. The authors compared a prototype Kinect‑based system to their existing CopyCat glove‑and‑accelerometer system by collecting 1,000 ASL phrases from both setups. The Kinect achieved 51.5 % (seated) and 76.12 % (standing) verification rates, comparable to the 74.82 % seated rate of the CopyCat system, indicating that with further tuning it could be a viable sign‑verification option.

Abstract

We investigate the potential of the Kinect depth-mapping camera for sign language recognition and verification for educational games for deaf children. We compare a prototype Kinect-based system to our current CopyCat system which uses colored gloves and embedded accelerometers to track children's hand movements. If successful, a Kinect-based approach could improve interactivity, user comfort, system robustness, system sustainability, cost, and ease of deployment. We collected a total of 1000 American Sign Language (ASL) phrases across both systems. On adult data, the Kinect system resulted in 51.5% and 76.12% sentence verification rates when the users were seated and standing respectively. These rates are comparable to the 74.82% verification rate when using the current(seated) CopyCat system. While the Kinect computer vision system requires more tuning for seated use, the results suggest that the Kinect may be a viable option for sign verification.