Evaluation of a Multilingual Synthetic Talking Faceas a Communication Aid for the Hearing Impaired

Abstract

The goal of the Synface project is to develop a multilingual synthetic talking face to aid the hearing impaired in telephone conversation. This report describes multilingual perceptual studies to characterise the potential gain in intelligibility derived from a synthetic talking head controlled by hand-annotated speech. Speech materials were simple Swedish, English and Dutch sentences typical of those used in speech audiometry. Speech was degraded to simulate in normal-hearing listeners the information losses that arise in severe-to-profound hearing impairment. Degradation was produced by vocoder-like processing using either two or three frequency bands, each excited by noise. 12 native speakers of each of the three languages took part in intelligibility tests in which each of the two degraded auditory signals were presented alone, with the synthetic face, and with a natural video of the face of the original talker. Intelligibility in the purely auditory conditions was low (7 % for the 2-band vocoder and 30 % for the 3-band vocoder). The average intelligibility increase for the synthetic face compared to no face was 20%, and was statistically highly reliable. The synthetic face fell short of the advantage of a natural face by an average of 18%. We conclude that the synthetic face in its current form is sufficient to provide important visual speech information.