CS Paper Radar

Multimodal Semantic Communication using Shared and Unique Representation Decomposition

Peng Yi, Ying-Chang Liang

cs.IT eess.SP ·submitted 2026-11-11 11:37 UTC · v1

Abstract

Semantic communication (SemCom) has emerged as a promising paradigm for bandwidth-constrained mobile and vehicular networks. This paper proposes a unified representation learning framework for multimodal SemCom (MSC), which encodes heterogeneous data into a compact semantic space for efficient transmission and source recovery. The framework addresses the multimodal semantic representation gap, which arises when at least one modality contains semantic information that cannot be fully inferred from the other. To preserve this information for reconstruction, the framework decomposes each representation into shared and modality-specific components. We formulate a mutual information objective for source reconstruction, cross-modal alignment of the shared components, and decoupling of the shared and modality-specific components. To enhance transmission efficiency, we introduce a shared knowledge base implemented as a compact codebook of semantic vectors learned during training. The shared components are transmitted digitally as codebook indices, while the modality-specific components are sent in analog form. We apply knowledge distillation during training, using a pretrained vision foundation model to guide the learning of quantized shared features. On image--text pairs over additive white Gaussian noise (AWGN) and Extended Vehicular A (EVA) channels, the proposed framework improves image reconstruction over conventional separate source-channel coding (SSCC) and representative multimodal and unimodal SemCom systems. Further evaluations show gradual degradation under channel-estimation mismatch, complementary effects of alignment, decoupling, and knowledge distillation on the shared representation, and alignment of shared features across modalities with separation of the modality-specific features.

Comment
This work has been submitted to the IEEE for possible publication

Closest papers

Nearest neighbours by meaning across the whole index.

Cooperative Multi-Task Semantic Communication for Joint Classification and Regression Tasks

eess.SP cs.LGeess.IVstat.ML ·2026-09-03 ·Ahmad Halimi Razlighi, Mohammad Siddiqur Rahman, Maximilian H. V. Tillmann, Edgar Beck, Armin Dekorsy
benchmark similarity 0.68
Abstract · This work has been submitted to the IEEE for possible publication

Multi-Task semantic communication (SemCom) prioritizes simultaneous execution of multiple tasks over bit-accurate reconstruction in future intelligent networks. In our prior work [1], we introduced the cooperative multi-task SemCom (CMT-SemCom) framework, in which the semantic encoder is divided into a common unit (CU) and multiple specific units (SUs) to facilitate cooperative multi-task processing. However, CMT-SemCom has been evaluated on homogeneous classification tasks on simplistic datasets, limiting its applicability to real-world perception systems. In this paper, we extend our CMT-SemCom to jointly handle heterogeneous classification and regression tasks on the complex Cityscapes dataset. We adopt the information maximization (InfoMax) principle so that it accommodates mixed discrete and continuous semantic variables. In particular, we benchmark the proposed framework against independent single-task training, a conventional task-agnostic digital transmission, and single-encoder multi-decoder SemCom. Additionally, we investigate the impact of CU capacity on joint task performance, providing design insights. Extensive evaluations demonstrate that CMT-SemCom significantly outperforms the benchmarks.

0.24

Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals?

cs.CV ·2026-10-27 ·Jialv Zou, Xiaotong Li, Zhengrong Yue, Xu Huang, Yaxin Luo +4 more
theorycode/datacommunity pick similarity 0.66
Abstract · Project page: https://doctor-james.github.io/FVQ-Page/

Discrete visual tokens represent vision in a discrete form analogous to language, offering a native interface for multimodal modeling under a shared next-token prediction objective. However, quantization error is widely assumed to cause unavoidable semantic loss, fundamentally limiting the semantic fidelity of discrete visual tokens and positioning continuous visual representations as an intrinsic upper bound. Using multimodal understanding as a proxy for semantic fidelity, we revisit this assumption and find quantization error unreliable, as discrete representations with substantially larger errors can achieve better understanding. Instead, the effective bit budget of discrete indices emerges as a key determinant of downstream performance. However, existing quantizers eventually saturate as additional codebooks contribute progressively less semantic information. As one approach to alleviating this saturation, we introduce Factorized Vector Quantization, which quantizes complementary, fixed-dimensional views of the visual feature with symmetric parallel codebooks. It continues to improve at larger bit budgets, ultimately matching continuous visual representations in multimodal understanding. Trained on industrial-scale multimodal data, our model achieves state-of-the-art performance among discrete-representation models, outperforming UniAR and Emu3-Chat by 22.8% and 59.2%, respectively, on a nine-metric average. To our knowledge, it is also the first discrete-representation model competitive with strong continuous-representation models such as Qwen3-VL, including on fine-grained OCR tasks. We further identify the key principles and a practical recipe enabling this competitiveness. These results and insights support the potential of discrete visual representations to underpin large-scale native multimodal foundation models. Code and models are available at https://github.com/hustvl/FVQ

0.56

Adaptive Semantic Communication with Residual Quantization for Resilient Vehicular Networks

cs.NI ·2026-10-07 ·Xuanhao Luo, Ruichen Gao, Sandeep Sreenivasan, Zhizhen Li, Yuchen Liu
similarity 0.65
Abstract

Vehicular networks require timely visual information exchange to support safety-critical applications such as cooperative perception and hazard awareness, yet transmitting high-dimensional camera data over bandwidth-limited and dynamic wireless links remains challenging. In this paper, we propose an adaptive semantic communication architecture for vehicular multicast that adjusts the amount of visual information transmitted according to current channel conditions, available communication resources, and latency requirements. Specifically, the visual encoder uses residual quantization to represent each image at multiple bitrate and reconstruction-quality levels. At runtime, the system selects an appropriate level based on the communication conditions, enabling it to trade visual fidelity for lower transmission overhead when network resources are limited. Context-adaptive entropy coding further compresses the quantized representations using offline-learned statistics, without requiring retraining for different channel conditions. This design allows the visual representation to be learned offline while the communication strategy is adapted online. Experimental results demonstrate competitive rate-distortion performance at ultra-low bitrates, improved delivery reliability and deadline satisfaction across diverse vehicular scenarios, and strong downstream perception performance, achieving up to 0.88 mAP50 while dynamically adapting to changing communication conditions.

0.15

Hub-Spectral Activation of Latent Multimodal Knowledge

cs.CV ·2026-09-15 ·Ying Guo, Haidong Chen, Linrui Xu, Xiaohao Liu, Chuancheng Shi +5 more
code/data similarity 0.65
Abstract · 30 pages, 9 figures, including appendices

Multimodal representation learning seeks shared representations for cross-modal retrieval and knowledge transfer. Hub-based binding reduces pairwise supervision costs, but separate hub connections cannot guarantee reliable alignment between modalities without direct joint training. We introduce Hub-Spectral Activation (HSA), a closed-form method for recovering and activating the hub-readable component of latent multimodal knowledge in frozen representations. We formalize this knowledge as source-induced cross-modal dependence and characterize the component determined by the second-order statistics of two trained hub edges. Under a second-order source model, we establish conditions for exact recovery of the complete source-induced relation and bound the dimension of its hub-readable component by the hub covariance rank. HSA composes and standardizes hub-edge statistics, extracts paired spectral directions, and combines reliability-weighted matching evidence with source-gated candidate resolution for bidirectional retrieval and prototype classification. HSA requires no target-pair supervision, gradient optimization, or backbone updates. Across 19 retrieval and 11 prototype-classification relations on ImageBind and LanguageBind, HSA raises mean bidirectional Recall@10 from 18.27% to 31.15% and mean macro Top-1 accuracy from 29.01% to 52.43%, respectively. Controlled analyses further identify valid hub-edge correspondence and leading spectral directions as key sources of retrieval gains, demonstrating the utility of latent multimodal knowledge beyond native similarity scores. Code and models are publicly available at https://github.com/Luo1Yan/HSA.

0.35

Federated Learning for Semantic Communications over Multiple Access Channels

cs.IT cs.DCcs.LGcs.NI+1 ·2026-10-27 ·Yalin E. Sagduyu, Tugba Erpek, Yi Shi
published similarity 0.65
Abstract

This paper presents a federated learning (FL) framework for semantic communication (SemCom) over the multiple access channel (MAC), where multiple distributed transmitters simultaneously convey semantic information to a common receiver using end-to-end learned semantic representations. Semantic feature extraction, compression, wireless transmission, and semantic inference are jointly optimized through FL-based SemCom. Unlike conventional SemCom systems that train users independently, simultaneous semantic transmissions over the MAC enable concurrent training of multiple transmitters within the same transmission interval, thereby reducing training overhead and channel usage. Federated semantic aggregation is incorporated with shared and personalized encoder components, allowing transmitters with limited local training data to benefit from semantic representations learned from transmitters possessing larger datasets while preserving local adaptation. Evaluated using the CIFAR-10 dataset over fading channels, the proposed approach maintains high semantic inference accuracy under shared wireless access and significantly improves performance under severe data imbalance. In the meantime, simultaneous semantic transmissions can significantly reduce training overhead while preserving robust semantic performance across distributed transmitters with heterogeneous local datasets. These results highlight the potential of combining FL and SemCom to support scalable and communication-efficient edge intelligence in future wireless networks.

0.28

Loss-Resilient Semantic Communication over Packet-Loss Networks at Extreme-Low Bandwidth

eess.IV ·2026-08-20 ·Shengshi Yao, Jincheng Dai, Sixian Wang, Guo Lu, Kai Niu +3 more
top venue similarity 0.65
Abstract · Accepted to appear in IEEE TMC. 14 pages, 15 figures, and 5 tables

In extreme-low bandwidth network scenarios, generative semantic codecs have emerged as promising solutions to reduce bandwidth cost for visual communications. However, these learned codecs are usually optimized solely for compression efficiency and thus not robust against transmission errors. Corruptions due to packet-loss among these highly compact generative latent representations often cause more critical degradation in fidelity and realism, intensified by the severe error propagation across the latent contexts and multi-step decoding process. In this paper, we propose ResiGLC, a novel loss-resilient generative latent coding framework designed for robust semantic communication over extreme-low bandwidth packet-loss networks. Motivated by the inherent goal-consistency between generation and compression, we sufficiently exploit the impressive in-context predictive capabilities of language models. Integrated with the masked learning strategy, our model supports arbitrary context modeling of latent codes, which could mitigate the error propagation and handle unpredictable packet loss patterns. At the receiver, a progressive resilient decoding pipeline is presented, which leverages both the contextual relationship of the latent codes and the multi-modal semantic prior in the generative latent space, separately. By jointly optimizing toward both compression efficiency and packet-loss resilience, our proposed progressive decoding mechanism offers graceful performance when dealing with dynamic packet losses. Through extensive experimental evaluations, we establish that under packet-loss network conditions, ResiGLC can effectively improve the loss-resilience in terms of perceptual fidelity and realism qualities with extreme-low bandwidth cost.

0.31

Relaxing Cross-Modal Correspondence for Multimodal Remote Sensing Pretraining

cs.CV cs.AI ·2026-11-10 ·Xianping Ma, Yu Zhang, Mingzhe Jiang, Wanxin Xie, Qing Zhu +3 more
similarity 0.64
Abstract · 16 pages, 9 figures, 13 tables

Multimodal remote sensing pretraining commonly relies on exact instance-level correspondence across modalities, implicitly treating strictly paired observations as a prerequisite for learning transferable representations. In this work, we revisit this assumption by relaxing cross-modal correspondence from the instance level to the semantic level. We introduce Semantic Pairing (SemPair), a pairing-relaxed pretraining strategy that replaces exact instance pairs with class-consistent cross-instance pairs. To characterize the effect of correspondence relaxation, we formulate a latent-factor model that separates cross-modal dependence into shared semantic structure and instance-specific correlation, showing that SemPair preserves the former while marginalizing instance-specific cross-modal correlations in expectation. We further introduce Mixed Pairing (MixPair) as a controlled interpolation between semantic and exact pairing to study correspondence granularity along a continuous spectrum. Extensive experiments consistently show that SemPair achieves transfer performance comparable to exact pairing (ExaPair), while MixPair reveals a non-monotonic relationship between downstream transfer and the instance-pairing ratio. These results demonstrate that exact instance correspondence is not a prerequisite for effective multimodal remote sensing pretraining and establish cross-modal correspondence granularity as an important design dimension for transferable multimodal representation learning. More broadly, this finding suggests a practical route toward scaling multimodal pretraining with reduced dependence on costly instance-aligned data.

0.26

Radio Echo Representation Learning for Semantic Sensing in Multi-Modal ISAC Systems

eess.SP ·2026-10-11 ·Guanping Shang, Anzheng Tang, Kwok Kit Chan, Shenghui Song
similarity 0.64
Abstract · This paper has been submitted to the IEEE trans journals

Semantic sensing extends integrated sensing and communication (ISAC) beyond physical parameter estimation by exploiting task-oriented representations from radio echoes. Nevertheless, existing multimodal sensing remains largely limited to vision-assisted radio physical parameter estimation or typically processes the visual and radio modalities separately, leaving radio representations underutilized for visual task enhancement. Feature-level fusion enables closer cross-modal interaction, while echo interference complicates the learning of user-specific radio representations and its semantic consistency with the visual entities. To address these challenges, this paper proposes a multimodal semantic sensing framework for orthogonal frequency-division multiplexing (OFDM)-ISAC systems. We propose a radio encoder to learn radio representations oriented toward visual localization from superimposed pilot echoes, together with Variational Radio-Visual Grounding (VRVG) to promote semantic consistency between the learned radio representations and visual spatial features. The radio representations are then formulated as semantic probes to highlight the corresponding entities in the visual scene through cross-modal fusion. Experiments using real-world RCooper dataset show that the resulting synergistic radio-visual embeddings enable accurate radio-visual user matching and significantly improve both user-specific visual localization and physical parameter estimation performance. The results substantiate the benefits of radio representation learning and semantically consistent fusion for multimodal semantic sensing.

0.30