A multimodal probabilistic model for gesture--based control of sound synthesis

TLDR

The paper proposes a multimodal approach to map gestures to sound in interactive music systems. It employs a multimodal HMM that jointly models gesture and sound parameters, enabling users to define gesture–sound relationships interactively and is implemented for controlling physical‑modeling sound synthesis. The resulting model effectively captures expressive gesture variations while maintaining a consistent gesture–sound relationship.

Abstract

In this paper, we propose a multimodal approach to create the mapping between gesture and sound in interactive music systems. Specifically, we propose to use a multimodal HMM to conjointly model the gesture and sound parameters. Our approach is compatible with a learning method that allows users to define the gesture--sound relationships interactively. We describe an implementation of this method for the control of physical modeling sound synthesis. Our model is promising to capture expressive gesture variations while guaranteeing a consistent relationship between gesture and sound.