Multimodal Interfaces: A Framework Based on Modality Appropriateness

TLDR

Human sensory systems specialize in different attributes, and while multimodal interfaces often redundantly present the same information, this can incur costs such as increased switching time and noise. This paper proposes a modality‑appropriateness framework that allocates information to the most suitable sensory channel and evaluates its benefits and drawbacks. The authors assessed the fit of 20 HCI information attributes to visual, auditory, and tactile modalities, defining a modality‑appropriateness metric to guide allocation of information chunks to the best‑matching channel.

Abstract

Our sensory modalities are specialized in perceiving different attributes of an object or event. This fact is the basis of the approach towards multimodal interfaces we describe in this paper. We rated the match between 20 possible information attributes (common in human computer interaction) and the visual, auditory and tactile sensory systems. We refer to this match as modality appropriateness. Preferably, an information chunk is allocated to the most appropriate modality. In situations in which information consists of several attributes, these may be allocated to different modalities. This approach is in contrast with the more common approach in which multimodality is implemented in an interface as a redundant presentation of the same information to two or more sensory modalities. This latter approach can be beneficial to solve risks of sensory overload and to make the interface accessible for people with a sensory challenge, but is not based on possible synergy between the senses. However, the supposed synergy may also involve costs, for example in terms of the time required to switch between modalities and in the introduction of additional noise in cross-modal comparisons compared to unimodal comparisons. We will discuss both the chances and the potential costs of applying the modality appropriateness framework.