CS Paper Radar

Look Before You Move: Multimodal Sensing-Aided Trajectory and Beamforming Design for 6DMA

Zeyu Hao, Wentao Yu, Lutz Lampe, Vincent W. S. Wong

eess.SP ·submitted 2026-11-04 05:36 UTC · v1

Abstract

Six-dimensional movable antenna (6DMA) technology allows a base station (BS) to adjust the three-dimensional (3D) positions and 3D rotations, i.e., the 6D poses, of its surfaces. Many existing 6DMA designs select the poses based on prior channel knowledge. However, the channels at a new pose can be estimated only after the surfaces have moved to this pose, and the limited movement speeds of the surfaces restrict the poses reachable at the next time step. To address these challenges, we propose a learning-based multimodal sensing-aided trajectory and beamforming design for multiuser 6DMA systems with mobile users and surrounding objects. The BS observes the users and surrounding objects with RGB images, LiDAR point clouds, and GPS positions, independently of the poses of the surfaces. Since a surface may need several time steps to reach a favorable pose, the BS designs a 6DMA trajectory over several future time steps and executes only its first step, following receding-horizon control. We propose a neural network, in which user-centric cross-modal attention and temporal self-attention modules relate each user to the scene and its sensing history, and surface-to-user attention relates each surface to the users. A projection layer keeps the poses feasible under the geometric and kinematic constraints, and a graph attention module allocates the transmit power along the pilot-based beamforming directions. Performance evaluations using the CARLA driving simulator and Sionna ray tracing (RT) demonstrate significant minimum-rate gains of the proposed design over fixed-position antennas. Ablation studies validate the contributions of multimodal sensing and the proposed attention modules to these gains. The results further highlight the importance of designing the trajectory several time steps ahead, which is more beneficial under tighter speed limits of the surfaces.

Comment
13 pages, 9 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Multimodal Learning for Beamforming Using Camera, LiDAR, and Radio-Frequency Pilots

eess.SP ·2026-09-18 ·Yinghan Li, Wei Yu
top venue similarity 0.67
Abstract

This paper proposes a multimodal learning framework that integrates camera images, LiDAR point clouds, and radio-frequency (RF) pilots to realize multi-user multiple-input single-output (MU-MISO) beamforming optimization in mixed line-of-sight (LoS) and non-line-of-sight (NLoS) environments. In LoS-dominant environments, beamforming can be designed by utilizing sensing information related to user locations. However, LoS paths cannot be guaranteed to exist for all users in realistic dynamic environments due to blockages. To characterize NLoS propagation, detailed three-dimensional (3D) information of surrounding obstacles is required. This paper integrates camera images and LiDAR points with RF pilots to capture detailed 3D geometric, object-class, and material information of users and surrounding objects. The obtained multimodal information provides useful cues for both direct LoS paths and reflected NLoS propagation, and provides rich channel-related information for beamforming design. Specifically, a bounding-box detection neural network is first used to extract two-dimensional spatial, object-class, and material information from camera images. The image-derived information then guides the selection of LiDAR points to obtain 3D geometric information of users and surrounding obstacles, and assigns object-class and material labels to the selected points. The image-guided LiDAR features, together with RF pilots, are jointly processed by a designed neural architecture to optimize the beamforming vectors. Simulation results show that the proposed method achieves a higher minimum rate than benchmarks.

0.63

Enabling Versatile Human Sensing through Wi-Fi Beamforming Feedback

eess.SP ·2026-11-12 ·Jingzhe Zhang, Peiliang Wang, Yitong Shen, Ning Wang, Mingzhe Chen +1 more
similarity 0.65
Abstract

Wi-Fi sensing enables contactless human sensing, while most existing systems rely on Channel State Information, which is accessible only on a limited range of Wi-Fi chipsets. In contrast, Beamforming Feedback Information (BFI) is standardized in modern Wi-Fi and readily accessible on a wide range of commodity devices, offering new opportunities for human sensing. In this paper, we present BeamLens, a Wi-Fi sensing system that extracts fine-grained spatial information from BFI to enable versatile human sensing. Specifically, we jointly estimate the angle of departure and relative time of flight from BFI to extract fine-grained spatial information and leverage two AP-STA links to capture complementary azimuth and elevation views. We further design a sparse physical antenna array and construct a uniform virtual antenna array to enhance spatial resolution. Moreover, we fuse complementary views through self-supervised learning, combining temporally aware cross-view contrastive learning and temporal masked reconstruction to learn a versatile backbone for diverse human sensing tasks. Extensive evaluations demonstrate that BeamLens achieves 90.16\% accuracy in human activity recognition, a mean per-joint position error of 8.64\,cm in 3D human pose estimation, and 70.64\% intersection over union in human silhouette estimation, while exhibiting strong generalization to unseen environments and subjects.

0.11

RadPose: ISAC-based RF 3D Human Pose Estimation

cs.CV cs.NI ·2026-11-01 ·Geo Jie Zhou, Kunzhe Song, Ruxin Lin, Huacheng Zeng
publishedcode/data similarity 0.62
Abstract · Accepted to WACV 2027

Human pose estimation under adverse lighting and occlusion conditions remains challenging for camera- and LiDAR-based systems. Although mmWave radars provide robustness to illumination and occlusion, they rely on dedicated hardware and licensed spectrum, limiting scalability. In this paper, we present RadPose, a learning-based monostatic integrated sensing and communication (ISAC) framework that enables mmWave communication devices to perform 3D human pose estimation. Unlike specialized radars, RadPose leverages monostatic full-duplex operation to extract channel impulse responses (CIRs) directly from standard OFDM communication signals, generating 3D radio point clouds without additional sensing hardware. To address the sparsity and noise of RF observations, RadPose employs a cross-frame transformer-based pose inference network with semantic-guided spatial attention and target-driven temporal attention for robust spatio-temporal aggregation. It further adopts a joint-aware inverse kinematics refinement module based on graph convolution to enforce skeletal topology, bone-length consistency, and anatomical plausibility. Extensive experiments demonstrate RadPose achieves competitive accuracy compared to state-of-the-art radar-based approaches while maintaining robust performance in unseen and occluded scenarios. The dataset and source code are available at https://github.com/Geo-Chou/RadPose.

0.46

PALIN: Pixel-Aligned LiDAR-NIR Multimodal 3D Object Detection with Benchmark

cs.CV ·2026-11-14 ·Yanan Zhang, Amit Vishnia, Chengbo Ai, Eleni Christofa, Runan Cheng +4 more
benchmark similarity 0.60
Abstract · 17 pages, including references and appendices

3D object detection is critical for intelligent transportation systems. LiDAR has been widely adopted for accurate geometric measurements; however, purely geometric representations remain ambiguous under sparse observations and occlusions, while cross-sensor multimodal fusion suffers from calibration and synchronization errors. To overcome these challenges, we leverage an Ouster LiDAR to construct PALIN (Pixel-Aligned LiDAR-NIR), a large-scale intrinsically pixel-aligned multimodal dataset jointly capturing geometry and NIR intensity, comprising approximately 8,000 annotated frames from static urban intersections and diverse mobile suburban scenes. Building upon this dataset, we propose CMambaFusion, a novel fusion framework consisting of a four-directional Cross-Mamba module for bidirectional cross-modal interaction and a Gated Concat-Mamba module for joint feature refinement. Extensive experiments demonstrate that PALIN enables effective multimodal learning, and CMambaFusion achieves superior accuracy-efficiency trade-offs over both LiDAR-only and representative fusion baselines.

0.18

EgoNeMo: Transferable Map of Pedestrian Dynamics via Egocentric LiDAR Scan

cs.LG cs.RO ·2026-09-05 ·Azusa Sawada, Allan Wang, Hideo Saito, Aaron Steinfeld
similarity 0.59
Abstract · 16 pages, 9 figures

This paper proposes a transferable Map of Dynamics (MoD) framework that generalizes to unknown environments using only egocentric 3D LiDAR point clouds to overcome the long-standing limitation of traditional MoD methods. While MoDs are essential for encoding human motion characteristics to enable accurate pedestrian trajectory prediction or safe robot navigation, traditional approaches suffer from site-specificity, requiring exhaustive trajectory accumulation at every new location. Extending recent advances in neural implicit modeling, our framework trains a continuous, LiDAR-based MoD estimator across diverse environments. To mitigate the inherent sparsity and temporal bias of real-world trajectory data, we introduce a position-balanced sampling strategy and a multi-task learning architecture that jointly predicts motion distributions and a spatial frequency score map. The latter is further augmented by visibility-aware losses to compensate for incomplete observation data. Comprehensive experiments demonstrate that our method effectively reconstructs underlying motion maps even in unknown locations from a single instantaneous LiDAR scan, despite highly sparse training data. Finally, we show that our improvements enhance the reliability of downstream trajectory prediction.

0.20

Pixel-UMI: Learning Environment-Aware Whole-Body Manipulation from Sensorized Human Demonstrations

cs.RO cs.AIcs.CVcs.LG+1 ·2026-11-02 ·Jason Jingzhou Liu, Tony Tao, Deepak Pathak, Ruslan Salakhutdinov
similarity 0.58
Abstract · Video results are available at https://pixel-robot.com/pixel-umi

Sensorized human demonstrations offer scalable data for mobile manipulation, but embodiment differences complicate policy learning. We present Pixel-UMI, a pipeline for learning environment-aware whole-body policies from human data. We first introduce Environment-Aware Retargeting (EAR), which combines reconstructed scene geometry with demonstrated head and gripper trajectories to determine robot posture and generate collision-aware joint and base targets. Delay-calibrated action assignment converts these trajectories into actions. However, kinematic differences can leave residual pose errors, creating viewpoint mismatches between human demonstrations and robot observations. We therefore introduce Retargeting-Error-Aware Masking (REAM), which uses these errors during training to reduce reliance on mismatched views at deployment. Finally, because estimated actions cannot fully capture contact-rich interactions, we collect on-policy human corrections through DAgger to improve performance. We evaluate Pixel-UMI on five challenging tasks spanning articulated-object manipulation, deformable-object handling, constrained-space reaching, whole-body collision avoidance, and contact-rich bimanual coordination, achieving 87.0% average task progress. Video results are available at https://pixel-robot.com/pixel-umi

0.46

SiGMA: Site-Grounded Multi-Modal ISAC Data Generation for Environment-Aware Communications

eess.SP cs.DB ·2026-11-16 ·Yijie Bian, Wei Guo, Kai Zhang, Xiao Ye, Shurui Xu +3 more
code/data similarity 0.58
Abstract

Learning-based multi-modal integrated sensing and communication (ISAC) requires data that associate environmental observations with the wireless response generated by the same site and dynamic scene state. Acquiring such data at new sites requires sensor calibration, coordinated measurements, and cross-modal association. We present site-grounded multi-modal ISAC data generation (SiGMA), which combines OpenStreetMap (OSM), Overture Maps, and procedural three-dimensional (3D) reconstruction to create georeferenced urban scenes. A shared scene geometry and time-indexed actor state link synchronized red-green-blue (RGB) images, light detection and ranging (LiDAR) point clouds, positioning records, frequency-modulated continuous-wave (FMCW) Radar in-phase and quadrature (IQ) samples, and ray-traced wireless channels. For configurable reflected propagation under direct-link blockage, SiGMA generates optional reconfigurable intelligent surface (RIS) component channels for phase-configuration evaluation across shared scene states. The evaluation results demonstrate that the framework improves zero-shot beam prediction performance on real DeepSense 6G measurements compared with an OSM-derived synthetic-data baseline. A camera-assisted RIS case study further demonstrates the generation of sensing-associated supervision for configurable reflected propagation. The dataset generation pipeline is available at https://github.com/Trans-cending/SiGMA-main.

0.39

RoboCap: A New Platform for Egocentric Robot Learning

cs.RO cs.AIcs.CVcs.HC ·2026-10-05 ·Qasim Ali, Nikash Bhardwaj, Kheri Hughes, Sam Cho, Ethan Lee +13 more
similarity 0.57
Abstract

Despite its promise for scaling robot learning, egocentric manipulation data is still scarce today. Collection at scale requires vertically integrating ergonomic hardware with centimeter-precise 3D algorithms, at a precision that has not been publicly demonstrated. To address this gap, we introduce RoboCap, a 250\,g six-camera dual-IMU hat designed for in-the-wild egocentric data capture, and the Grounded API, a suite of device-agnostic 3D algorithms tuned for RoboCap. In this report, we demonstrate how hardware, calibration, and 3D algorithms interact to achieve state-of-the-art performance on the public benchmarks: our SLAM across diverse settings and rigs, our depth estimation on egocentric settings, and our hand tracking when adapted to third-party devices.

0.16