Silent Speech Eyewear Interface: Silent Speech Recognition Method using Eyewear with Infrared Distance Sensors

Abstract

As eyewear devices such as smart glasses become more common, it is important to provide input methods that can be used at all times for such situations and people. Silent speech interaction (SSI) has the potential to be useful as a hands-free input method for various situations and people, including those who have difficulty with voiced speech. However, previous methods have involved sensor devices that are difficult to use anytime and anywhere. We propose a method for SSI that involves using an eyewear device equipped with infrared distance sensors. The proposed method measures facial skin movements associated with a speech from the infrared distance sensor mounted on an eyewear device and recognizes silent speech commands by applying machine learning to the time-series sensor data. The proposed method was applied to a prototype system including a sensor device consisting of eyewear and ear-mounted microphones to measure the movements of the cheek, jaw joint, and jaw. From the evaluation results, F-measure was 0.90 with five speech commands and 0.83 with ten longer speech commands, which indicates the feasibility of the proposed method as a simple hands-free input interface. Our study provides the first wearable sensing method that can easily apply SSI functions to eyewear devices.