SmartKom: Towards Multimodal Dialogues with Anthropomorphic Interface Agents

Abstract

SmartKom is a multimodal dialogue system that combines speech, gesture, and facial expressions for input and output. SmartKom provides an anthropomorphic and affective user interface through its personification of an interface agent. Understanding of spontaneous speech is combined with video-based recognition of natural gestures and facial expressions. One of the major scientific goals of SmartKom is to design new computational methods for the seamless integration and mutual disambiguation of multimodal input and output on a semantic and pragmatic level. SmartKom is based on the situated delegation-oriented dialogue paradigm, in which the user delegates a task to a virtual communication assistant, visualized as a life-like character on a graphical display. SmartKom is a multilingual system that analyses and generates German and English utterances. We describe the SmartKom architecture, the use of an XML-based mark-up language for multimodal content, and the most distinguishing features of the fully operational SmartKom 2.0 system. 1.