A Unified Multi-Task Semantic Communication System for Multimodal Data

TLDR

Task‑oriented semantic communications have achieved significant performance gains, yet deep neural networks must be updated or stored separately when tasks change or multiple tasks are required. This work proposes a unified deep learning‑enabled semantic communication system (U‑DeepSC) that can serve many different tasks with multiple modalities of data. U‑DeepSC employs a vector‑wise dynamic scheme that adjusts the number of transmitted symbols per task and channel, a lightweight feature‑selection module that hierarchically drops redundant feature vectors to accelerate inference, and a unified codebook that transmits only indices of task‑specific features. Simulation results demonstrate that U‑DeepSC matches the performance of task‑specific semantic communication systems while substantially reducing transmission overhead and model size.

Abstract

Task-oriented semantic communications have achieved significant performance gains. However, the employed deep neural networks in semantic communications have to be updated when the task is changed or multiple models need to be stored for performing different tasks. To address this issue, we develop a unified deep learning-enabled semantic communication system (U-DeepSC), where a unified end-to-end framework can serve many different tasks with multiple modalities of data. As the number of required features varies from task to task, we propose a vector-wise dynamic scheme that can adjust the number of transmitted symbols for different tasks. Moreover, our dynamic scheme can also adaptively adjust the number of transmitted features under different channel conditions to optimize the transmission efficiency. Particularly, we devise a lightweight feature selection module (FSM) to evaluate the importance of feature vectors, which can hierarchically drop redundant feature vectors and significantly accelerate the inference. To reduce the transmission overhead, we then design a unified codebook for feature representation to serve multiple tasks, where only the indices of these task-specific features in the codebook are transmitted. According to the simulation results, the proposed U-DeepSC achieves comparable performance to the task-oriented semantic communication system designed for a specific task but with significant reduction in both transmission overhead and model size.