Word predictability after hesitations: a corpus-based study

Abstract

Asks whether lexical hesitations in spontaneous speech tend to precede words that are difficult to predict. We define predictability in terms of both transition probability and entropy, in the context of an N-gram language model. The results show that the transition probability is significantly lower at hesitation transitions, and that this is attributable to both the following word and the word history. In addition, the results suggest that fluent transitions in sentences with a hesitation elsewhere are significantly more likely to contain out-of-vocabulary words and novel word combinations. Such findings could be used to improve statistical language modeling for spontaneous speech applications.