Shake, rattle, and roll : Getting immersed in multisensory, interactive music via broadband networks

Abstract

Broadband Internet (transmission rates more than a gigabit per second) enables bidirectional real-time transmission of multiple streams of audio, video, and motion data with latency dependent on distance plus network and processing delays. In this article we describe a new immersive multisensory environment recently constructed at McGill University, designed for network-based communication for music performance coordinated between remote sites, potentially over great distance. The system's architecture allows participants to experience the music with greatly enhanced presence through the use of multiple sensors and effectors and high-resolution multimodal transmission channels. Up to 24 channels of audio, digital video, and four channels of vibration can be sent and received over the network simultaneously, allowing a number of diverse applications such as remote music teaching, student auditions, jam sessions and concerts, recording sessions, and postproduction for remotely-captured live events. The technical and operational challenges of this undertaking are described, as well as potential future applications.