Toward Semantic Communications: Deep Learning-Based Image Semantic Coding

TLDR

Semantic communications can dramatically reduce transmitted data while preserving essential information, yet most research has focused on text using NLP techniques. This study aims to extend semantic communications to image data, which carries richer semantics and is more bandwidth‑sensitive. The authors propose RL‑ASC, a reinforcement‑learning framework that encodes images by extracting category, spatial, and visual semantic concepts with a convolutional encoder, allocates bits adaptively to maximize a rate‑semantic‑perceptual reward, and reconstructs images via a GAN‑based decoder that fuses local and global features through attention. Experiments show that RL‑ASC preserves task‑related information, discards less important data, and achieves noise‑robust, visually pleasant, and semantically consistent reconstructions at low bit rates.

Abstract

Semantic communications has received growing interest since it can remarkably reduce the amount of data to be transmitted without missing critical information. Most existing works explore the semantic encoding and transmission for text and apply techniques in Natural Language Processing (NLP) to interpret the meaning of the text. In this paper, we conceive the semantic communications for image data that is much more richer in semantics and bandwidth sensitive. We propose an reinforcement learning based adaptive semantic coding (RL-ASC) approach that encodes images beyond pixel level. Firstly, we define the semantic concept of image data that includes the category, spatial arrangement, and visual feature as the representation unit, and propose a convolutional semantic encoder to extract semantic concepts. Secondly, we propose the image reconstruction criterion that evolves from the traditional pixel similarity to semantic similarity and perceptual performance. Thirdly, we design a novel RL-based semantic bit allocation model, whose reward is the increase in rate-semantic-perceptual performance after encoding a certain semantic concept with adaptive quantization level. Thus, the task-related information is preserved and reconstructed properly while less important data is discarded. Finally, we propose the Generative Adversarial Nets (GANs) based semantic decoder that fuses both locally and globally features via an attention module. Experimental results demonstrate that the proposed RL-ASC is noise robust and could reconstruct visually pleasant and semantic consistent image in low bit rate condition.