CS Paper Radar

Channel Gains to Captions: Task-Unified Multi-Level RF Sensing with Vision-Language Models

Tianyu Hu, Zhiren Gong, Haowei Cui, Shuai Wang, Samson Lasaulce, Lingxiang Li, Wassim Hamidouche, Zhi Chen, Merouane Debbah

eess.SP ·submitted 2026-08-31 14:07 UTC · v1
published

Abstract

This letter investigates a task-unified multi-level radio-frequency (RF) sensing framework driven by vision-language models (VLMs). Existing RF sensing methods rely on task-specific designs and provide only partial environmental information, limiting their ability to handle emerging 6G applications. To address this, we propose a generative formulation for RF sensing, where millimeter-wave (mmWave)/terahertz (THz) channel gains are mapped to captions describing multi-level environmental semantics. The framework solves this problem through a complementary design for RF-environment semantic bridging, where a VLM is fine-tuned to leverage its multimodal representations and prompt-conditioned semantic generation capabilities. Hence, different sensing tasks are specified through textual prompts, enabling the framework to handle diverse tasks in a unified manner. For fine-tuning, we introduce prompt-routed low-rank adaptation (LoRA) experts to achieve level-aware adaptation. Simulation results show that, compared with baselines, our framework achieves superior performance with a broader semantic scope, and enables task-unified sensing beyond predefined tasks. Under an unseen sensing requirement, it achieves an average F1-score improvement of 0.17 over the most competitive variant.

Closest papers

Nearest neighbours by meaning across the whole index.

Can Language Models Understand mmWave Data? Benchmarking Large Language Models for mmWave Radar-Based Human Understanding

cs.AI ·2026-08-14 ·Jeongwan Shin, Jaehyeon Kim, Donguk Ko, Jaeho Choi
top venuebenchmark similarity 0.67
Abstract · Accepted to CVPR 2026 Findings

Large language models (LLMs) have shown remarkable reasoning and generative capabilities, motivating their use as universal reasoning engines for perception. While modern approaches such as vision-language models (VLMs) have attempted to incorporate reasoning capabilities into visual sensing, the integration of LLMs with the millimeter-wave (mmWave) modality-despite its unique advantages under low light and occlusion-remains largely unexplored. The principal bottlenecks stem from the scarcity of radar language pairs, severe cross-dataset heterogeneity, and the absence of a foundational mmWave encoder. We address this gap through a minimal textualization interface that serializes each mmWave point cloud into concise natural language, allowing off-the-shelf LLMs to operate in a question answering (QA) setting. Building on this, we present mmWave-QA, the first benchmark for language-conditioned mmWave human perception. mmWave-QA aggregates heterogeneous public mmWave datasets and harmonizes them via calibration-aware preprocessing and global taxonomy alignment, while providing natural language QA. Spanning six scenarios and five QA tasks, the benchmark enables standardized evaluation across diverse mmWave hardware and experimental conditions, establishing a foundation for scalable research on mmWave-LLM integration. We further evaluate and analyze LLMs on our mmWave-QA, highlighting their zero-shot reasoning potential for radar perception, as well as their robustness under visual degradation.

0.76

Radio Echo Representation Learning for Semantic Sensing in Multi-Modal ISAC Systems

eess.SP ·2026-10-11 ·Guanping Shang, Anzheng Tang, Kwok Kit Chan, Shenghui Song
similarity 0.60
Abstract · This paper has been submitted to the IEEE trans journals

Semantic sensing extends integrated sensing and communication (ISAC) beyond physical parameter estimation by exploiting task-oriented representations from radio echoes. Nevertheless, existing multimodal sensing remains largely limited to vision-assisted radio physical parameter estimation or typically processes the visual and radio modalities separately, leaving radio representations underutilized for visual task enhancement. Feature-level fusion enables closer cross-modal interaction, while echo interference complicates the learning of user-specific radio representations and its semantic consistency with the visual entities. To address these challenges, this paper proposes a multimodal semantic sensing framework for orthogonal frequency-division multiplexing (OFDM)-ISAC systems. We propose a radio encoder to learn radio representations oriented toward visual localization from superimposed pilot echoes, together with Variational Radio-Visual Grounding (VRVG) to promote semantic consistency between the learned radio representations and visual spatial features. The radio representations are then formulated as semantic probes to highlight the corresponding entities in the visual scene through cross-modal fusion. Experiments using real-world RCooper dataset show that the resulting synergistic radio-visual embeddings enable accurate radio-visual user matching and significantly improve both user-specific visual localization and physical parameter estimation performance. The results substantiate the benefits of radio representation learning and semantically consistent fusion for multimodal semantic sensing.

0.30

Wireless Foundation Models: State-of-the-Art and Open Challenges

eess.SP cs.AI ·2026-09-04 ·Alonso M. Pacheco Huachaca, Juan J. Rodriguez Rodriguez, Ahmed Aboulfotouh, Nelson L. S. da Fonseca, Carlos A. Astudillo +1 more
survey similarity 0.58
Abstract

Wireless foundation models (WFMs) have emerged as a promising approach for learning reusable representations from large-scale wireless data and adapting them to downstream tasks. However, the rapidly growing literature remains fragmented across modalities, pretraining objectives, architectures, adaptation strategies, and evaluation protocols, making it difficult to assess progress toward broadly transferable models. This survey provides a systematic analysis of WFMs for physical-layer applications. We first introduce the main WFM design components, including pretraining, backbone architectures, and downstream adaptation. We then organize the literature into five physical-layer task families: signal recognition and demodulation, channel representation learning, RF sensing and localization, beam management, and spectrum sensing and monitoring, while separately examining multi-task PHY models. Across these categories, we analyze how existing models are pretrained, adapted, and evaluated, with particular attention to downstream task diversity and the distinction between in-distribution, partial-shift, and out-of-distribution transfer. Our analysis shows that current WFMs provide increasing evidence of reusable wireless representations, but this evidence varies considerably across task families and evaluation settings. Differences in datasets, modalities, architectures, pretraining objectives, adaptation protocols, and distribution shifts make it difficult to determine which design choices drive transfer and generalization. We conclude by identifying open directions for improving data availability, evaluation rigor, generalization, efficient adaptation, and real-world deployment, providing a unified framework for understanding the current WFM landscape and the requirements for developing more reusable foundation models for future physical-layer wireless systems.

0.40

ControlRadio: Prompt-Driven Controllable Diffusion for Cross-Modal Radio Map Generation

cs.CV ·2026-08-10 ·Kangjun Liu, Xiying Pan, Shuhang Zhang, Xiang Xiang, Ke Chen +1 more
similarity 0.58
Abstract · 17 pages, 9 figures, 8 tables

Radio maps describe how wireless signals propagate across space and are essential for wireless communication, sensing, and network planning. However, constructing accurate radio maps traditionally requires either dense measurements or computationally expensive physical simulations, which limits scalability and real-time deployment. Recent advances in generative artificial intelligence offer a promising alternative, but existing approaches lack fine-grained control and physical consistency when applied to real-world wireless environments. Here we present \textbf{ControlRadio}, a controllable generative framework that produces radio maps from natural-language descriptions and environmental layouts, including building structures and transmitter locations. Joint semantic and spatial conditioning enables interpretable, propagation-plausible generation, while a controlled latent prior and layout-aware conditioning improve stability and structural consistency. Extensive experiments demonstrate that ControlRadio achieves state-of-the-art accuracy and strong generalization across diverse urban scenarios, while reducing computation time by more than four orders of magnitude compared with conventional simulation-based methods. Such results suggest a new paradigm for scalable and controllable wireless environment modeling, with broad implications for next-generation communication systems and data-driven radio sensing.

0.10

Task-disentangled Low-Rank Adaptation for Versatile Audio-visual Multi-modal Learning Tasks within a Unified Framework

cs.MM ·2026-08-25 ·Hanyu Xuan, Mengqi Zhang, Junjun Mao, Fei Wang, Kun Li +3 more
similarity 0.58
Abstract

Inspired by human multi-modal perception, Audio-Visual Multi-Modal Learning (AVMML) integrates auditory and visual information to leverage complementary cross-modal cues, enabling more robust and comprehensive scene perception. Existing studies predominantly tackle each AVMML task in isolation, which stands in stark contrast to humans' unified cognitive capacity for handling versatile perception. However, naive joint training across multiple AVMML tasks often suffers from mutual interference, arising from the intricate inter-task relationships. To address this, we propose a unified framework that simultaneously accommodates versatile AVMML tasks. Specifically, benefiting from powerful representation and generalization capabilities of large language models, we design a task-disentangled Low-Rank Adaptation (LoRA) mechanism that enables dynamic integration of both task-specific and task-shared knowledge, thereby facilitating effective multi-task collaboration. The proposed task-disentangled LoRA comprises three components: a task-general low-rank matrix, task-specific modulation matrices, and cross-task collaboration experts, which respectively capture universal audio-visual knowledge, decouple task-specific pattern, and exploit inherent inter-task correlations. By unifying explicit collaboration from both model and task perspectives, our approach not only surpasses existing unified audio-visual models across multiple AVMML tasks, but also outperforms most task-specific models on certain AVMML tasks.

0.44

Bridging Semantics and Signals: An LLM-Driven MoE Framework for Radio Map Prediction

eess.SP ·2026-11-07 ·Qi Yang, Xinyue Jiang, Tong Li, Chanchan Zhu, Shaolong Gu +1 more
similarity 0.57
Abstract

High-fidelity radio map prediction is essential for wireless network optimization, yet existing single-network models frequently fail to generalize across diverse multi-band and multi-terrain scenarios due to rigid parameter constraints. We introduce LLM-MoE, a framework that employs a divide-and-conquer Mixture-of-Experts strategy to dynamically allocate modeling tasks to matched expert sub-networks. Our approach features a heterogeneous expert pool for characterizing varied physical propagation behaviors, combined with an LLM-driven semantic extractor that provides global physical awareness. A cross-modal fusion module integrates these macroscopic semantics with local spatial features for adaptive routing. Experimental results on the SpectrumNet dataset show that LLM-MoE outperforms top-tier baselines, achieving a 10.0\% reduction in Mean Squared Error (MSE) and 9.7\% in Normalized Mean Squared Error (NMSE), while ensuring robust generalization across heterogeneous environments.

0.18

Instruction-Conditioned Electromagnetic Spectrum Understanding via Budget-Adaptive Signal Tokenization

cs.AI eess.SP ·2026-10-08 ·Lei Zhai, Zhihao Chang, Shuyuan Yang, Zhixi Feng
similarity 0.57
Abstract

Electromagnetic spectrum monitoring increasingly requires flexible analysis beyond task-specific recognition and detection. Multimodal large language models offer a unified interface, but extending vision-language models (VLMs) to raw I/Q signals requires tokenization that balances fidelity against a strict budget. For signals, dense encoding causes token costs to grow with observation length, whereas fixed-resolution compression may discard short-duration or localized signal evidence. Thus, we propose \textbf{BATok}, a budget-adaptive signal tokenizer that adjusts token capacity to the input length while allocating that capacity according to the signal content. BATok constructs candidate representations from signal-derived features using lightweight multi-resolution branches, then combines a local energy prior with learnable queries to resample these representations into compact signal tokens. The number of tokens adapts to the input length while remaining strictly bounded. The resulting tokens are projected into the language embedding space of VLMs. We further introduce \textbf{EMSpec-Instruct}, a multimodal instruction dataset aligning raw I/Q signals, waterfall images, and language supervision for modulation recognition, structured detection, and language-conditioned signal grounding. Experiments show that BATok learns effective signal representations and achieves competitive performance across all tasks.

0.24

SemISAC: Semantic Integrated Sensing and Communications

eess.SP ·2026-09-16 ·Xiaoqi Zhang, J. Andrew Zhang, Zhongqin Wang, Chang Liu, Weijie Yuan +2 more
similarity 0.57
Abstract

Conventional integrated sensing and communications (ISAC) systems primarily integrate communications and sensing through shared physical resources, without explicitly exploiting task-relevant semantic information. To move beyond such physical-level integration, we propose semantic ISAC (SemISAC), a general framework that unifies semantic communication (SemCom) and semantic sensing (SemS) to convey source meaning and acquire environmental meaning. Specifically, the transmitter combines source semantics and sensing task information with available side information to design the shared waveform and allocate radio resources, while the receiver-side communication and sensing task decoders recover the source meaning and infer the required environmental information, respectively. We also provide an information-theoretic interpretation to characterize the relationship between physical and task-relevant information and the resulting semantic trade-off in SemISAC. Building on this framework, we formulate the general SemISAC design problem and propose two realization methods, namely end-to-end (E2E) SemISAC optimization and modular SemISAC optimization. As a concrete realization, we apply modular SemISAC optimization to jointly design a learnable time-frequency (TF) precoder in an orthogonal frequency-division multiplexing (OFDM) system for representative SemCom and SemS tasks. Simulation results demonstrate that the proposed realization reduces sensing semantic distortion under a given communication requirement and achieves a more favorable communication-sensing trade-off than baseline designs.

0.40