ComputingComputer ScienceKnowledge RepresentationNatural Language ProcessingSensor FusionConversational User InterfacesData AbstractionMultimodal AnalysisMultimodal CommunicationMultimodal CompositionMultimodal InteractionAbstractionAbstraction LevelsData Stream ProcessingMultimodal FusionMultimodal Interface DesignMultimodal InterfacesMultimodal User InterfacesSemantic Inference
In recent years, multimodal interfaces have gained momentum as an alternative to traditional WIMP interaction styles. Existing multimodal fusion engines and frameworks range from low-level data stream-oriented approaches to high-level semantic inference-based solutions. However, there is a lack of multimodal interaction engines offering native fusion support across different levels of abstractions to fully exploit the power of multimodal interactions. We present Mudra, a unified multimodal interaction framework supporting the integrated processing of low-level data streams as well as high-level semantic inferences. Our solution is based on a central fact base in combination with a declarative rule-based language to derive new facts at different abstraction levels. Our innovative architecture for multimodal interaction encourages the use of software engineering principles such as modularisation and composition to support a growing set of input modalities as well as to enable the integration of existing or novel multimodal fusion engines.
17
Richard A. Bolt · 1980 · 1.4K citations · Full text
Paul Dietz, Darren Leigh · 2001 · 1.1K citations
Engineering, Electrical Engineering, Mechatronics Engineering +17
Richard A. Bolt · ACM SIGGRAPH Computer Graphics · 1980 · 904 citations