A Toolkit for Creating and Testing Multimodal Interface Designs

Abstract

Designing and implementing applications that can handle multiple recognition-based interaction technologies such as speech and gesture inputs is a difficult task. IMBuilder and MEngine are the two components of a new toolkit for rapidly creating and testing multimodal interface designs. First, an interaction model is specified in the form of a collection of finite state machines, using a simple graphical tool (IMBuilder). Then, this interaction model can be tested in a multimodal framework (MEngine) that automatically performs input recognition (speech and gesture) and modality integration. Developers can build complete multimodal applications without concerning themselves with the recognition engine internals and modality integration. Furthermore, several interaction models can be rapidly tested in order to achieve the best use and combination of input modalities with minimal implementation effort.