Visual Prosody and Speech Intelligibility

TLDR

The study recorded 3‑D head and face motion and acoustics of Japanese speakers, then used these recordings to create realistic talking‑head animations that could manipulate head motion while keeping other visual and acoustic features constant, and tested them in a speech‑in‑noise perception task. The study found that natural head motion conveys linguistic cues correlated with pitch and amplitude, and that including realistic head motion in talking‑head animations improves syllable identification in noisy speech, indicating a direct role for nonverbal gestures in speech perception.

Abstract

People naturally move their heads when they speak, and our study shows that this rhythmic head motion conveys linguistic information. Three-dimensional head and face motion and the acoustics of a talker producing Japanese sentences were recorded and analyzed. The head movement correlated strongly with the pitch (fundamental frequency) and amplitude of the talker's voice. In a perception study, Japanese subjects viewed realistic talking-head animations based on these movement recordings in a speech-in-noise task. The animations allowed the head motion to be manipulated without changing other characteristics of the visual or acoustic speech. Subjects correctly identified more syllables when natural head motion was present in the animation than when it was eliminated or distorted. These results suggest that nonverbal gestures such as head movements play a more direct role in the perception of speech than previously known.