Cross-Modal Collaborative Communications

Abstract

It is well known that cross-modal services, including audio, video, and haptic signals, will inevitably become the mainstream of multimedia applications. However, because there are significant differences among these three kinds of signals in terms of transmission delay, jitter, and reliability, how to effectively transmit and process them is an extremely challenging problem. Unlike the traditional tactile Internet which mainly focuses on haptic, this work proposes a collaborative communications mechanism by exploring the temporal, spatial, and semantic relevance of cross-modal signals. On one hand, we design a content-driven scheduling scheme to guarantee high-quality cross-modal services by leveraging the spatio-temporal transmission characteristics. On the other hand, we develop a complete cross-modal signal restoration, reconstruction, and rendering architecture through semantic-based signal fusion and sharing. Importantly, we demonstrate that the proposed mechanism can improve the immersive experience of the user substantially and considerably.