CS Paper Radar

SoM-MTM: Synesthesia of Machines (SoM)-Driven Masked Token Model for Cooperative Perception over Packet Loss Channel

Haozhen Li, Rongqing Zhang, Xiang Cheng

eess.IV ·submitted 2026-08-13 13:50 UTC · v1

Abstract

To support the large-scale and heterogeneous visual cooperative perception (CP) demands in next-generation mobile networks, intelligent and efficient sensory data transmission is a critical challenge. Under the emerging convergence of communication networks and agentic artificial intelligence (AI), existing research emphasizes utilizing end-to-end neural networks to simplify communication modules, which has shown promising potential for CP. However, these studies are still limited to specific channel models, cooperation modes, and perception tasks, failing to fully leverage powerful visual processing approaches to enhance universality. To address this, we propose a Synesthesia of Machines (SoM)-driven Masked Token Model, referred to as SoM-MTM, as a plug-and-play paradigm for generic visual CP. Inspired by masked image modeling methods such as MAE, it possesses great perceptual context learning capabilities to recover distorted features over packet loss channels, thereby improving information carrying efficiency. Building upon Swin Transformer, SoM-MTM further embeds prior masked information through an External Routing MoE mechanism, maximally repairing and enhancing environmental perception features during cooperation. Comprehensive experimental results confirm that SoM-MTM can consistently enhance perception performances on various tasks, especially strong generalization to unseen scenarios, while maintaining favorable model cost and scalability.

Closest papers

Nearest neighbours by meaning across the whole index.

SemRD-V2X: Closure-Guided Communication with Bounded Inference for Cooperative Perception

cs.AI cs.IT ·2026-09-28 ·Hu Xu, Chun Li, Siyuan Qiu, Zeyan Li, Jianfeng Xu
similarity 0.63
Abstract

Vehicle-to-Everything (V2X) cooperative perception improves 3-D detection by sharing intermediate features, but dense remote features may repeat context that the ego agent can infer locally. Most communication-efficient designs optimize masks or codes empirically, leaving a more basic question open: which remote evidence is indispensable given the receiver's own observation? We introduce a closure-fidelity perspective on ego conditioned remote perception. Under a finite deductive abstraction and explicit conditions, its rate--distortion function decomposes over an irredundant core, and the exact zero-distortion rate becomes $P_A H(π_A)$. This analysis suggests a concrete design principle: transmit compact evidence and recover derivable context with bounded receiver-side inference. Guided by this principle, SemRD-V2X is an operational neural proxy that combines exact-budget BEV support selection, pointwise channel compression, and masked shared-weight reconstruction before standard fusion. Experiments on simulated V2XSet and real-world DAIR-V2X validate the resulting design. In a controlled five-run V2XSet comparison against a locally reproduced V2X-ViT-v1 baseline on one Tesla V100, SemRD-V2X reduces the analytical feature payload by $26.6\times$ while improving AP@0.5/AP@0.7 by 4.13/8.57 points, with 3.81\% additional mean compute latency. These results position closure fidelity as both an analytical lens and an actionable design principle for communication-efficient cooperative perception.

0.13

Disentangling Channel Semantics in Vision Transformers via Token Decorrelation and Composition-Aware Modulation

cs.CV ·2026-11-07 ·Daeun Kim, Hyesong Choi, Hyejin Park, Dongbo Min
top venuecode/data similarity 0.62
Abstract · Accepted at NeurIPS 2026. 32 pages, 12 figures

Multi-channel images encode heterogeneous channel semantics aligned over a shared spatial structure, which requires models to jointly capture globally shared structure and channel-specific variations. Recent multi-channel Vision Transformers attempt to address this challenge by augmenting the [CLS] token with memory tokens to organize multi-channel representations under varying channel compositions. However, we find that these context tokens collapse onto a few dominant channels under high inter-channel redundancy, producing redundant representations and a biased global summary that fails to reliably support composition-aware channel interpretation. We propose MuCa-ViT (Multi-Channel Composition-Aware Vision Transformer), a unified framework that addresses these limitations through two coupled mechanisms. Factorial Token Learning (FTL) enforces token-level decorrelation among context tokens, encouraging the [CLS] token to capture globally shared semantics while memory tokens preserve complementary channel-specific cues. Channel Composition-Aware Modulation (CAM) uses the FTL-refined [CLS] as a semantic anchor to generate sample-wise modulation signals, which enable composition-aware interpretation of channels within a unified backbone rather than relying on static channel embeddings. Experiments on JUMP-CP, CHAMMI, and So2Sat show that MuCa-ViT outperforms prior multi-channel ViTs across all three benchmarks, and the magnitude of improvement aligns with the inter-channel redundancy of each dataset ($ρ\in [0.19, 0.77]$), validating the diagnostic analysis that motivates our design. Our code is available at https://github.com/kimdanni/MuCa-ViT .

0.55

Mobile Embodied Agent Network (MEAN): Integrated Framework, Enabling Technologies, and Task-Driven Deployment

cs.NI eess.SY ·2026-10-19 ·Zhaoyang Zhang, Lingxiao Sun, Weijie Zhou, Chunhui Feng, Zhaohui Yang +6 more
survey similarity 0.61
Abstract

Mobile networks are evolving from human-centered, data-oriented communication infrastructures toward machine- and/or agent-dominant, task-oriented platforms that are to serve the rapidly increasing networked machines and embodied agents. In order to fulfill the newly arose complex task objectives, the framework of future mobile network is to be innovated to reflect and facilitate the intrinsic interplay and integration of the underlying processes of perception, communication, computation, decision, action, and feedback, etc. This survey establishes the mobile embodied agent network (MEAN) as a task-oriented networking paradigm in which networking and distributed intelligence are jointly organized around closed-loop embodied task execution. We first establish a unified MEAN framework that explains how network resources and intelligent functions jointly support embodied task execution, and extend its performance indicators from conventional communication metrics to embodied task outcomes and cross-layer system effects. With this framework, we analyze three complementary enabling mechanisms that form a cognition-interaction-adaption loop. Large artificial intelligence (AI) models provide high-level cognition, token communication connects this cognition with distributed agents through shared semantic representations, and reinforcement learning closes the loop by adapting system operation to network, task, and physical feedback. We further investigate the bidirectional interplay between MEAN and emerging technologies, showing how they mutually extend system capabilities and reshape task-oriented network design. Finally, we discuss open challenges and future research directions toward the practical realization of MEAN over real joint communication and computation platform.

0.23

Token-Oriented Semantic Communication with Pretrained Vision Transformers

eess.SP cs.AIcs.CVcs.LG ·2026-08-26 ·Jiwoong Im, Minwoo Kim, Jaeho Lee, Yo-Seb Jeon, Yongjune Kim
similarity 0.58
Abstract

Token communications realize the semantic communication principle at the granularity of transformer tokens, providing a promising direction for client--server collaborative inference in resource-constrained edge systems. However, directly transmitting token embeddings presents two practical challenges: substantial communication cost and limited interoperability across model-specific token embedding spaces. To address these challenges, we propose a \emph{token-oriented} semantic communication framework. In this framework, token-level task relevance determines which compressed image latents are transmitted, enabling token-granular transmission without directly transmitting token embeddings. The framework is modular, coordinating three pretrained components---a lightweight client-side vision transformer (ViT), a learned image compression (LIC) model, and a large server-side ViT---without end-to-end training. The key enabler is the one-to-one spatial alignment between ViT patch tokens and the LIC latent vectors, which allows token-level task relevance to directly determine which latent vectors are transmitted. Building on this alignment, token-aligned LIC selectively transmits task-relevant latents, layer-selective attention rollout estimates token relevance from a selected range of attention layers in a single forward pass, and surrogate token substitution adapts the frozen server model by optimizing a single learnable token. Experiments on ImageNet show that the proposed framework achieves a more favorable rate--accuracy trade-off than recent semantic communication schemes, hand-crafted codecs, and task-agnostic LIC models.

0.23

Thinking in Tokens, Talking in Bits: A Practical Interface for Token Communication

eess.SP cs.ITeess.IV ·2026-09-14 ·Chanho Park, Bumsu Park, Soonhee Kwon, Sangrim Lee, Namyoon Lee
similarity 0.58
Abstract · 7 pages, 4 figures

Advanced artificial intelligence models think in tokens; contemporary communication systems carry bits. The direct way to bridge this gap is to transmit tokens, but that makes a model-specific representation part of the air interface, coupling the endpoints through a shared tokenizer, codebook, and often a neural transceiver. We take a different route: keep bits in the payload and let tokens control how those bits are generated and protected. The resulting token-bit interface transition aligns task-side tokens with source- and channel-coding units, translates token relevance into codec controls, and preserves the induced priority order across the coding chain. We instantiate it for image classification, where a vision transformer scores the task relevance of each image region from its attention maps: those scores steer block-wise JPEG rate allocation, then group the compressed bits for protection at different polar-code rates. The payload remains an explicit, reconstructable bitstream recovered by a correspondingly configured decoder. Over-the-air experiments on a software-defined radio testbed show improved accuracy--latency tradeoffs over separate source-channel coding, performance competitive with far more memory-intensive neural joint source-channel coding, and graceful degradation under channel mismatch. Token communication, then, need not transmit tokens explicitly; what it needs is an interface through which tokens determine how bits are communicated.

0.14

Feature-Aware Token Attack for Compression-Triggered Stealthy Failures in Large Vision-Language Models

cs.CV ·2026-09-30 ·Shilinlu Yan, Bowen Chen, Yuechen Zhang, Zhenhong Zhou, Li Sun +1 more
similarity 0.57
Abstract · 29 pages including references and appendices, 10 figures. Submitted to ICLR 2027

Visual-token compression improves the efficiency of large vision-language models, but can expose failures that full-token evaluation misses. We study adversarial images that preserve full-token correctness yet induce errors after compression, even when both inference paths succeed on the clean image. Creating such failures is challenging because perturbing token importance can also damage the visual content needed for full-token inference. We propose Feature-Aware Token Attack (FATA), which couples attention suppression with cosine-based feature preservation on a fixed set of salient clean-image tokens. In the primary LLaVA-1.5-7B setting, FATA uses only vision-encoder gradients, without access to the deployed compressor, token budget, or downstream task. Across four visually dependent task subsets and four compressors under a controlled reconstruction protocol, FATA achieves SR = 96.3% full-token accuracy retention and CBR = 22.1% conditional blinding, compared with 89.8% and 15.7% for CAA. Ablations support the role of both objectives in balancing compressed-path failure against full-token preservation. FATA also has the lowest measured detection rate among four attacks across three evaluated detectors at a 5% false-positive rate. These findings motivate assessing adversarial robustness jointly across full-token and compressed inference.

0.22

Beyond Token Stacking: Multi-scale Visual Modulation and Adaptive Fused-feature Calibration for MLLMs

cs.CV ·2026-11-16 ·Ye Xiao, Shesheng Song, Yi Ren, Shasha Li, Bin Ji +3 more
top venue similarity 0.56
Abstract

Current Multimodal Large Language Models (MLLMs) have advanced rapidly and follow two main paradigms: one involves concatenating visual and textual tokens for input into Large Language Model (LLM), while the other injects visual knowledge into Feed-Forward Networks (FFNs) of the LLM. However, the former introduces a large number of redundant visual tokens, substantially increasing computational overhead. The latter reduces sequence length but removes images from the input stage entirely, which may hinder the activation of image-related knowledge. Moreover, naively injecting visual knowledge into the FFNs could induce activation distribution shifts due to cross-modal semantic discrepancies. To address these challenges, we develop Beyond Token Stacking (BTS), a novel visual enhancement framework for MLLMs. Specifically, we propose a Multi-scale Visual Modulation (MVM) mechanism that injects multi-scale visual information at the input stage via modulation without increasing the token sequence length. Furthermore, we develop an Adaptive Fused-feature Calibration (AFC) module that performs per-channel affine calibration to decouple visual injection from distribution alignment in the FFN layers, improving multimodal fusion stability while preserving language priors. Experiments show that our approach achieves state-of-the-art performance.

0.42

Modality-Decoupled Federated Learning for Privacy-Preserving Embodied Intelligence in 6G

eess.SP cs.AI ·2026-09-09 ·Zhuodong Liu, Xiangyu Li, Chunhong Yuan, Hongyang Du, Bodong Shang +3 more
published similarity 0.56
Abstract · This article has been accepted for publication in IEEE Wireless Commnunications Magazine

Sixth-generation (6G) wireless networks are expected to provide a key infrastructure for large-scale embodied intelligence, where heterogeneous robots collaborate through low-latency connectivity, edge intelligence, and distributed sensing. Vision-language-action (VLA) models offer a foundation by integrating visual perception, language understanding, and action generation into a unified closed-loop policy. However, training and adapting VLA models to distributed robotic agents introduce challenges in privacy protection, communication efficiency, and model heterogeneity. Existing federated learning (FL) methods overlook the intrinsic differences among vision, language, and action pathways in parameter scale, privacy exposure, update dynamics, and tolerance to compression or perturbation. To address this issue, this article proposes FedMVLA, a modality-decoupled FL framework for privacy-preserving embodied intelligence in 6G networks. FedMVLA incorporates three mechanisms: modality-aware federated aggregation (MAFA), modality-aware privacy allocation (MAPA), and modality-aware communication compression (MACO), together with a modality-sliced transport design that routes the precision-critical action stream through a protected ultra-reliable low-latency slice. A case study on federated robotic manipulation over the Third Generation Partnership Project (3GPP)-based wireless substrate, covering fading, co-channel interference, and malicious jamming, shows that FedMVLA achieves an 84.8% task success rate, exceeds FedAvg by 22.2 percentage points, sustains a widening margin when scaling to 128 clients across eight cells, and reduces the schedule-averaged per-client uplink model-update payload by 95.6% (approximately 96%), while keeping the 95th percentile (p95) of the round-critical uplink completion time near 1.5s.

0.34