CS Paper Radar

Goal-Oriented Communications for Physical AI: Design and Testbed

Shutong Chen, Wenkai Zhang, Adnan Aijaz, Miao Guo, Yansha Deng

cs.RO eess.IV ·submitted 2026-09-14 17:16 UTC · v1

Abstract

Physical AI relies on frequently-updated, latency-sensitive video stream to perceive, reason, and interact with the physical world, resulting in strict latency requirements with much higher data volumes that existing 5G networks cannot support. Goal-oriented communication (GoC) offers as a promising approach to solve this challenge by transmitting only task-relevant semantic representations. However, existing GoC frameworks were mainly evaluated in the simulations while their effectiveness has never been validated in a practical deployment of physical AI application. In this work, we develop an end-to-end GoC testbed for Physical AI, which connects a PiPER robot arm equipped with an RGB-D camera and a 5G modem to an NVIDIA Jetson AGX Orin edge server through a 5G OpenAirInterface network. We propose and implement three GoC frameworks that transmit 3D bounding boxes, 2D scene graphs, and 3D scene graphs, as three types of semantic representations, respectively. They share the common functional modules designed for closed-loop Physical AI applications, including semantic extraction, full stack 5G transmission, language model inference, digital twin validation, and robotic control. Extensive experiments on our testbed show that our GoC frameworks reduce the task completion time by up to 52.6% and improve task success probability by up to 45%, compared to the traditional framework that periodically transmits the raw image data. These results validate the practical effectiveness of our GoC framework and pave the way for efficient and reliable Physical AI applications over future 6G networks. Project website: https://sites.google.com/view/goc-physical-ai-testbed.

Comment
Submitting to IEEE for potential publications

Closest papers

Nearest neighbours by meaning across the whole index.

A Hierarchical Onboard Vision-Language-Navigation Framework for Aerial Robots in Complex Environments

cs.RO cs.AI ·2026-11-18 ·Yuze Wu, Mo Zhu, Xingxing Li, Yuheng Du, Zhaoqi Wang +6 more
top venue similarity 0.65
Abstract · 20 pages, 14 figures

Unmanned Aerial Vehicles (UAVs) are expected to achieve semantic reasoning, intelligent decision-making, and long-horizon navigation under strict onboard constraints. While recent foundation-model-driven approaches have shown promising progress toward this goal, existing methods remain limited by challenges in 3D data availability, trajectory feasibility, long-horizon reasoning, and computational efficiency in complex environments. To bridge these gaps, we propose VLN-AR, a hierarchical vision-language-navigation (VLN) framework for aerial robots. VLN-AR decouples multimodal reasoning and low-level control, enabling structured prediction of 3D pose commands and navigation intentions that are translated into dynamically feasible actions. To mitigate data scarcity and domain gaps, we introduce a fully automated 3D Gaussian Splatting (3DGS)-based pipeline to generate scalable photorealistic UAV navigation data. A progressive three-stage training strategy further enhances grounding, scene reasoning, and complex navigation. In addition, a lightweight cognitive agent improves long-horizon reasoning via memory-aware task decomposition. Finally, we design onboard optimization techniques, including token sparsification and model lightweighting, achieving 7.1x-9.4x speedup while maintaining high navigation performance on resource-limited UAV platforms. Experiments show great improvements over existing methods in both simulation and real-world UAV navigation. Project website: https://wyz000.github.io/VLN-AR/

0.47

Wireless Edge-Assisted Adaptive Latent Reasoning for Vision-Language-Action Models in Physical AI

cs.RO cs.NI ·2026-11-14 ·Fatih Temiz, Melike Erol-Kantarci
benchmark similarity 0.65
Abstract · 6 pages, 6 figures, submitted to IEEE ICC

Vision-language-action (VLA) models enable embodied artificial intelligence (AI) by translating visual observations and language instructions into robotic actions. Emerging use cases in physical AI are expected to drive the need for efficient inference and communication, as the computational demands of the VLA models pose challenges for real-time control with limited onboard resources. Test-time scaling through latent reasoning allows these models to refine hidden representations without generating explicit chain-of-thought (CoT) tokens, but introduces a trade-off between task success, inference latency, and energy consumption. In this paper, we address the efficient collaborative inference under a realistic communication scenario. We propose a wireless edge-assisted framework for adaptive latent reasoning (WEALR) that augments onboard computation with edge computing. A decision-making agent jointly selects the execution location and latent-reasoning convergence threshold. We formulate this joint adaptation as a constrained Markov decision process (CMDP). A Lagrangian-constrained deep reinforcement learning (CDRL) approach, namely trust region policy optimization Lagrangian (TRPO-Lag), learns to balance task performance and resource consumption under varying wireless conditions and task requirements. We evaluate the framework using LIBERO manipulation tasks, profiled VLA inference costs, and ray-traced wireless channels. Experimental results demonstrate that TRPO-Lag achieves up to 23% higher feasibility-weighted task success rate compared to benchmark CDRL methods.

0.09

The Robot Data Factory

cs.RO ·2026-09-15 ·Sami Haddadin, Ivan Laptev, Ian Reid, Dezhen Song, Cesare Stefanini +16 more
benchmarkposition similarity 0.64
Abstract

Physical AI requires more than increasingly large robot datasets: intelligent robots acquire knowledge through continuous interaction with the physical world. We argue that the defining scientific resource of Physical AI is therefore not raw robot data alone, but robot experience - physically grounded interaction whose observations, actions, embodiment, context, and outcomes preserve the perception-action-consequence loop. We introduce the Robot Data Factory (RDF), a mission-driven infrastructure and methodology for continuously generating, validating, benchmarking, and reusing such experience. RDF organizes heterogeneous robots and environment-specific training grounds through reproducible missions, skill curricula, synchronized multimodal sensing, external ground truth, an agentic robot network, data pipelines, and living benchmarks. Rather than treating datasets as static end products, RDF implements a closed Deploy-Measure-Learn-Repeat cycle in which validated physical experience supports world models, vision-language-action models, embodied policies, digital twins, and subsequent robot deployment. We further formalize robot experience and its quality, introduce a mission-task-skill-episode-dataset-benchmark-capability hierarchy, and derive quantitative scaling laws and an algorithmic synthesis procedure connecting robot fleet size, sensor rates, storage, learning representations, tokenization, training compute, inference, and latency to Embodied-AI cluster requirements. The framework is instantiated in three complementary physical training grounds for domestic, environmental, and energy applications. RDF thus reframes robot data generation as a continuous scientific production process and provides a pathway toward reproducible, scalable, and eventually federated infrastructure for Physical AI.

0.20

Embodied Zero-Shot Task Planning via Hardware-Aware LLM-VLM Integration for Grounded Robotic Manipulation

cs.RO cs.AI ·2026-10-08 ·Lonnie Tankersley, Noorbakhsh Amiri Golilarz, Soroush Korivand
published similarity 0.63
Abstract · 6 pages, 5 figures, 1 table. Accepted and presented at the 25th IEEE International Conference on Machine Learning and Applications (ICMLA 20…

Large language models can translate natural-language instructions into plausible manipulation plans, but robotic execution requires physically grounded decisions that respect robot embodiment, workspace geometry, reachability, collision constraints, and grasp feasibility. This paper presents a hardware-aware large language model-vision-language model (LLM-VLM) manipulation framework for a simulated Franka Research 3 robot in NVIDIA Isaac Sim. The framework separates symbolic reasoning from metric execution: an API-hosted LLM planner receives a structured scene summary and generates only symbolic task information, including the object to manipulate, target relation, approach preference, and objects to avoid. A local Qwen3-VL 8B VLM verifier supports semantic confirmation of ambiguous objects and open-container surrogates. Both the LLM planner and the VLM verifier are prohibited from generating robot coordinates, grasp poses, keepout dimensions, gripper commands, trajectory waypoints, or success labels. Instead, these decisions are handled by deterministic grounding and execution modules that use the robot model, scene geometry, MoveIt 2 collision checking, inverse kinematics, grasp validation, release verification, and final visual assessment. Execution feedback is returned only as symbolic context for replanning. The framework is evaluated through simulated digital-twin trials involving pickup, occluded-source access, keepout reasoning, relation placement, and sequential manipulation tasks.

0.27

Impact of Data Compression on Downstream AI Tasks: A Study using Teleoperated Driving over 5G

cs.NI ·2026-09-21 ·Qixin Zhang, Steven Sleder, Xinyue Hu, Faaiq Bilal, Wei Ye +1 more
highly cited similarity 0.63
Abstract · Published in IEEE CQR 2024

Teleoperation, such as remote driving, is considered as a key use case of 5G and Next-Generation (NextG) networks. In this context, robots, autonomous vehicles, or other autonomous agents transmit sensor data over mobile networks to edge or cloud servers, where AI systems collaborate with human operators to provide situational awareness and enable remote control. In the case of teleoperated driving, vehicles are equipped with an array of cameras and LiDAR devices, which can generate 100s Mbps (megabits per second) of data. As shown in existing measurement studies, such data volumes far exceed the \emph{uplink} capacity of currently deployed 5G networks, especially when multiple vehicles compete for radio resources. Data compression is thus imperative. In this paper, we explore the impact of sensor data compression on the performance of downstream AI tasks running in edge/cloud servers, which are crucial to alert human operators for safe teleoperation. Using object recognition and semantic segmentation as two example AI tasks, we study how data compression affects the performance of these two AI tasks using unimodal (video or LiDAR) and multi-modal (video+LiDAR) data. We find that lossy data compression generally decreases the performance of AI tasks. The performances of these AI tasks exhibit differing degrees of sensitivity based on the types of data sources and levels of compression. We also empirically identify an optimal trade-off point for the multi-modal vision tasks.

0.63

Implicit-Reasoning-Enabled Elastic Semantic Communications for AI-Agent Networks

eess.SY cs.AI ·2026-11-16 ·Kaiwen Yu, Gang Wu, Jiancun Fan, Saviour Zammit
similarity 0.63
Abstract

AI-agent networks introduce new communication demands driven by proactive interactions and autonomous task execution. Existing adaptive semantic communication schemes primarily adjust semantic representations and resource configurations within predefined processing scales and capability sets, which may be insufficient for evolving task requirements. To address this gap, we introduce elastic semantic communications (ESC), a task-driven framework that extends semantic adaptation to the runtime execution and participation of task-supporting capabilities. ESC comprises two complementary dimensions: vertical elasticity adjusts the execution intensity of existing capabilities, while horizontal elasticity reconfigures the participating capability set. The framework connects these dimensions through the expected marginal value of additional information and computation. To realize ESC, we propose an implicit-reasoning-enabled device--edge mechanism that supports variable-depth local inference through shared-parameter state updates and edge-side continuation through wireless transfer of intermediate reasoning states. Experiments on Sudoku-Extreme demonstrate that the benefits of additional inference depend on wireless conditions, while task-aware edge invocation can preserve task accuracy with substantially lower state-transfer overhead.

0.18

Objectives as Policies: Vision-Language Model Cost Programs for Contact-Rich Manipulation

cs.RO ·2026-11-13 ·Zihao Lu, Trong Gia Hung Nguyen, Fuquan Wang, Yifan Zhu
similarity 0.63
Abstract · 8 pages, 3 figures. Project website: https://zihaolu001.github.io/oap-project/

Vision-language models (VLMs) have shown strong semantic and spatial reasoning capabilities in 3D environments. Recently, connecting VLMs with model-predictive controllers (MPC) has emerged as a paradigm for combining strong semantic reasoning with physics-based planning. However, connecting these two for contact-rich robotic manipulation requires a task representation that bridges the two and captures how physical interactions can be reasoned by VLMs. In this work, we introduce Objectives as Policies (OaP), a framework that connects a pretrained VLM to a shared sampling-based MPC through an editable objective program. Each program specifies a sequence of stages for completing a task, combining running and terminal costs from a cost library with stage completion conditions. Given a natural-language task instruction and scene observation, we build a simulation-based digital twin through real-to-sim, and the VLM generates an objective program that is executed by the MPC in the simulation. The program is iteratively updated by the VLM based on MPC rollouts, before being deployed in the real world. We evaluate our framework in both simulation and the real world on four tabletop manipulation tasks, and show that it outperforms strong baselines. Additional videos and examples are available on the project website: https://zihaolu001.github.io/oap-project/

0.23

Show-Harness: Just a VLM Agent Can Play Robots

cs.RO cs.AIcs.CVcs.MM ·2026-09-09 ·Yanzhe Chen, Zechen Bai, Zhijun Cao, Wenzheng Zeng, Kevin Qinghong Lin +5 more
code/datahighly cited similarity 0.62
Abstract · Project website: https://showlab.github.io/Show-Harness

Foundation vision-language models (VLMs) exhibit broad intelligence about the world, yet translating this intelligence into robot control remains challenging. We present Show-Harness, an Embodied Harness that enables VLMs to "play" robots through a compact semantic interface linking intent to action. Show-Harness exposes discrete semantic action units that VLMs can naturally reason over, while embodiment-specific interpreters deterministically ground them into local robot actions, keeping the VLM directly responsible for fine-grained physical decisions. Through the same interface, Show-Harness demonstrates the feasibility of (1) directly unlocking closed-source frontier VLMs for zero-shot robot control, and (2) adapting small-scale open-source VLMs for low-cost deployment with just a few GPU-hours of fine-tuning. We further develop GUMI (GUI Manipulation Interface), which extends the same semantic action space to GUI-based demonstration collection, allowing humans and agents to "play" robots across embodiments without specialized teleoperation hardware. Extensive experiments show that Show-Harness-equipped VLM agents generalize robustly across tasks, embodiments, and environments, outperforming representative agentic and VLA paradigms. These results suggest that the right interface can unlock substantial embodied capability from foundation VLMs, without requiring additional model capacity or costly embodiment-specific pretraining.

0.48