Combining Speech, Gaze, and Micro-gestures for the Multimodal Control of In-Car Functions

Abstract

Modern cars are already incredibly smart environments today due to the sheer number of sensors and processors packed into a small space. Likewise, new technologies in human-computer interaction increasingly find their way inside, e.g. eye tracking, speech interaction and gesture recognition. The support of new modalities is promising a reduction of driver distraction and a better handling of an increasing number of functions offered by in-vehicle systems. With multiple modalities to choose from, which can be combined arbitrarily via multimodal fusion, drivers can make a free choice depending on the demands of the situation and their preferences. Our paper presents a prototype in-car system that allows car features (like turning lights and windows) to be controlled by combinations of speech, gaze, and micro-gestures. We propose an interaction concept, sketch our architecture based on a domain-independent multimodal dialogue platform, and draw some first conclusions on the outcome.