The timing of speech-accompanying gestures with respect to prosody

Abstract

The question of how and whether the body movements that accompany speaking are timed with respect to the speech has often been studied, and investigators have reached different conclusions depending on the types of gestures and aspects of prosody attended to. The ToBI system for labeling pitch accents (phrase-level prosodic prominences) and intonational phrase boundaries, which provides a well-defined inventory of prosodic elements, was used to label several sound files from videotaped lectures in English. A particular type of gesture, i.e., discrete sharp rapid movements that reach a perceptually salient end point, was separately labeled for syllable location in the visual display of the same lecture samples. Preliminary analysis showed a strong correlation between this type of ‘‘stroke-like’’ gesture of the head or hands and pitch accented syllables. For example, for one speaker of Australian English, 168 of 195 strokelike gestures (86%) occurred with a pitch-accented syllable. If these observations from coarse-grained temporal labeling are confirmed by the frame-by-frame labeling now under way, it will suggest that the study of speech-accompanying gestures can provide evidence for the prosodic structure of spoken utterances, and raise the possibility that a complete model of speech production planning should include a gestural component.