Evaluation of a Silent Speech Interface Based on Magnetic Sensing and Deep Learning for a Phonetically Rich Vocabulary

Abstract

To help people who have lost their voice following total laryngectomy,
\nwe present a speech restoration system that produces
\naudible speech from articulator movement. The speech articulators
\nare monitored by sensing changes in magnetic field caused
\nby movements of small magnets attached to the lips and tongue.
\nThen, articulator movement is mapped to a sequence of speech
\nparameter vectors using a transformation learned from simultaneous
\nrecordings of speech and articulatory data. In this work,
\nthis transformation is performed using a type of recurrent neural
\nnetwork (RNN) with fixed latency, which is suitable for realtime
\nprocessing. The system is evaluated on a phoneticallyrich
\ndatabase with simultaneous recordings of speech and articulatory
\ndata made by non-impaired subjects. Experimental results
\nshow that our RNN-based mapping obtains more accurate
\nspeech reconstructions (evaluated using objective quality metrics
\nand a listening test) than articulatory-to-acoustic mappings
\nusing Gaussian mixture models (GMMs) or deep neural networks
\n(DNNs). Moreover, our fixed-latency RNN architecture
\nprovides comparable performance to an utterance-level batch
\nmapping using bidirectional RNNs (BiRNNs).