A Study of the Feasibility and Effectiveness of Dual-Modal Information Presentations

Abstract

Multimodal interfaces with both visual and auditory output are becoming important, especially for applications using small-screen displays and for user access under mobile conditions. The research presented here investigated the feasibility of simultaneously presenting distinct textual information through both visual and auditory channels by examining two multimodal interfaces with irrelevant or relevant auditory information. These interfaces were intended to study two problems: (a) Can users attend to and process additional information delivered through the auditory channel during a typical Web-browsing process, and (b) what are the effects of information overlap between the visual and auditory channels? Controlled experiments were conducted to evaluate these two questions. The findings suggest that users can attend to auditory information while visually browsing textual information and that information overlap may reduce distraction. These findings have implications for the design of multimodal interfaces for small-screen mobile applications.