CS Paper Radar

Multimodal Learning for Beamforming Using Camera, LiDAR, and Radio-Frequency Pilots

Yinghan Li, Wei Yu

eess.SP ·submitted 2026-09-18 01:17 UTC · v1
top venue

Abstract

This paper proposes a multimodal learning framework that integrates camera images, LiDAR point clouds, and radio-frequency (RF) pilots to realize multi-user multiple-input single-output (MU-MISO) beamforming optimization in mixed line-of-sight (LoS) and non-line-of-sight (NLoS) environments. In LoS-dominant environments, beamforming can be designed by utilizing sensing information related to user locations. However, LoS paths cannot be guaranteed to exist for all users in realistic dynamic environments due to blockages. To characterize NLoS propagation, detailed three-dimensional (3D) information of surrounding obstacles is required. This paper integrates camera images and LiDAR points with RF pilots to capture detailed 3D geometric, object-class, and material information of users and surrounding objects. The obtained multimodal information provides useful cues for both direct LoS paths and reflected NLoS propagation, and provides rich channel-related information for beamforming design. Specifically, a bounding-box detection neural network is first used to extract two-dimensional spatial, object-class, and material information from camera images. The image-derived information then guides the selection of LiDAR points to obtain 3D geometric information of users and surrounding obstacles, and assigns object-class and material labels to the selected points. The image-guided LiDAR features, together with RF pilots, are jointly processed by a designed neural architecture to optimize the beamforming vectors. Simulation results show that the proposed method achieves a higher minimum rate than benchmarks.

Closest papers

Nearest neighbours by meaning across the whole index.

Look Before You Move: Multimodal Sensing-Aided Trajectory and Beamforming Design for 6DMA

eess.SP ·2026-11-04 ·Zeyu Hao, Wentao Yu, Lutz Lampe, Vincent W. S. Wong
similarity 0.67
Abstract · 13 pages, 9 figures

Six-dimensional movable antenna (6DMA) technology allows a base station (BS) to adjust the three-dimensional (3D) positions and 3D rotations, i.e., the 6D poses, of its surfaces. Many existing 6DMA designs select the poses based on prior channel knowledge. However, the channels at a new pose can be estimated only after the surfaces have moved to this pose, and the limited movement speeds of the surfaces restrict the poses reachable at the next time step. To address these challenges, we propose a learning-based multimodal sensing-aided trajectory and beamforming design for multiuser 6DMA systems with mobile users and surrounding objects. The BS observes the users and surrounding objects with RGB images, LiDAR point clouds, and GPS positions, independently of the poses of the surfaces. Since a surface may need several time steps to reach a favorable pose, the BS designs a 6DMA trajectory over several future time steps and executes only its first step, following receding-horizon control. We propose a neural network, in which user-centric cross-modal attention and temporal self-attention modules relate each user to the scene and its sensing history, and surface-to-user attention relates each surface to the users. A projection layer keeps the poses feasible under the geometric and kinematic constraints, and a graph attention module allocates the transmit power along the pilot-based beamforming directions. Performance evaluations using the CARLA driving simulator and Sionna ray tracing (RT) demonstrate significant minimum-rate gains of the proposed design over fixed-position antennas. Ablation studies validate the contributions of multimodal sensing and the proposed attention modules to these gains. The results further highlight the importance of designing the trajectory several time steps ahead, which is more beneficial under tighter speed limits of the surfaces.

0.29

Resource-Efficient 3D Shape and Orientation Estimation of Road Users using Roadside RGB Camera and 3D Radar

cs.CV cs.AI ·2026-10-13 ·Shiva Agrawal, Savankumar Bhanderi, Gordon Elger
published similarity 0.62
Abstract · Accepted at the 18th IEEE Symposium Sensor Data Fusion: Trends, Solutions and Applications (SDF 2026), Bonn, Germany

End-to-end deep learning pipelines for camera-radar 3D object detection are well established in driver assistance systems and autonomous vehicles; however, adapting them to roadside-mounted infrastructure presents significant challenges. On vehicles, sensor heights, pitch angles, and relative geometries remain largely uniform, allowing end-to-end models to generalize effectively. On the contrary, roadside sensor installations vary drastically in height, tilt, and field of view across locations, rendering end-to-end models prone to failure and difficult to adapt without massive location-specific datasets. Hence, instead of designing a complex end-to-end pipeline for direct 3D bounding box estimation, a low-cost, resource-efficient two-stage approach is proposed for real-time 3D shape and orientation estimation. The first stage utilizes any off-the-shelf, state-of-the-art method to extract 2D camera bounding boxes and associated 3D radar points for each road user. The proposed second stage then processes these inputs to construct 3D oriented bounding boxes where radar data determines 3D position, predefined templates define size for each road user, and a tiny neural network predicts a site-specific and optimized yaw angle. The algorithm is successfully validated at a curved road with a junction for pedestrians, bicycles, and cars, where it achieves 25 frames per second on an NVIDIA Jetson ORIN for real-time applications.

0.36

RLG-TPV: Radar- and LiDAR-Guided Tri-Perspective View Fusion for Camera-Radar 3D Object Detection

cs.CV ·2026-08-29 ·Ahmet Mete Dokgoz, A. Enes Doruk, Hasan F. Ates
similarity 0.62
Abstract

Tri-Perspective View (TPV) representations describe 3D scene structure through top, side, and front feature planes, but existing TPV lifting is primarily camera-based, leaving the depth of sampled image evidence ambiguous along projected camera rays. We propose RLG-TPV, a multimodal TPV framework for camera-radar 3D object detection in which radar and training-time LiDAR provide complementary geometric guidance during representation construction. A ray-guided deformable-attention lift weights sampled image features using LiDAR-supervised camera depth probabilities and radar frustum occupancy, while radar additionally refines the depth distribution before lifting. Because conventional radar provides limited elevation information, LiDAR-derived class-occupancy targets supervise the side and front planes during training; the corresponding heads are removed at inference, so deployment requires only cameras and radar. For temporal aggregation, Doppler-guided temporal fusion aligns past features using a motion field anchored by measured radar radial velocity, with gating that limits warping in regions without supported motion. An RCS-aware radar scatter further allows radar evidence to spread over spatial neighborhoods conditioned on radar cross section. On the nuScenes validation set, RLG-TPV achieves 0.4981 mAP and 0.5959 NDS, reducing orientation and velocity error by 31.9\% and 30.7\% relative to the published CRN baseline. Ablation studies show that ray-level geometric guidance is a major contributor to the final performance.

0.09

PALIN: Pixel-Aligned LiDAR-NIR Multimodal 3D Object Detection with Benchmark

cs.CV ·2026-11-14 ·Yanan Zhang, Amit Vishnia, Chengbo Ai, Eleni Christofa, Runan Cheng +4 more
benchmark similarity 0.61
Abstract · 17 pages, including references and appendices

3D object detection is critical for intelligent transportation systems. LiDAR has been widely adopted for accurate geometric measurements; however, purely geometric representations remain ambiguous under sparse observations and occlusions, while cross-sensor multimodal fusion suffers from calibration and synchronization errors. To overcome these challenges, we leverage an Ouster LiDAR to construct PALIN (Pixel-Aligned LiDAR-NIR), a large-scale intrinsically pixel-aligned multimodal dataset jointly capturing geometry and NIR intensity, comprising approximately 8,000 annotated frames from static urban intersections and diverse mobile suburban scenes. Building upon this dataset, we propose CMambaFusion, a novel fusion framework consisting of a four-directional Cross-Mamba module for bidirectional cross-modal interaction and a Gated Concat-Mamba module for joint feature refinement. Extensive experiments demonstrate that PALIN enables effective multimodal learning, and CMambaFusion achieves superior accuracy-efficiency trade-offs over both LiDAR-only and representative fusion baselines.

0.18

LiDAR-Derived Surface Priors for Multimodal Sensing-Assisted NLoS Beam Search in Indoor 60-GHz Networks

eess.SY ·2026-08-18 ·Amod Ashtekar, Dalton Davis, Rafaela Lomboy, Omar Ibrahim, Raj Sai Sohel Bandari +1 more
similarity 0.61
Abstract

Highly directional 60-GHz Internet-of-Things (IoT) links can exploit naturally occurring indoor surfaces to sustain connectivity under blockage. Identifying viable non-line-of-sight (NLoS) paths, however, can require extensive RF beam training. This paper investigates whether LiDAR can reduce this search overhead by providing a surface-aware prior without assuming a direct mapping between optical return and mmWave reflection. The proposed framework uses LiDAR-derived geometry and return statistics to rank candidate propagation directions, while RF measurements remain responsible for final beam selection. The experimental validation is organized in three stages to separate descriptor robustness, cross-modal association, and beam-search performance. Controlled LiDAR measurements first quantify how geometric and radiometric surface descriptors vary with acquisition geometry. Matched LiDAR and 60-GHz measurements in an L-shaped corridor then determine whether these descriptors are associated with the measured surface-mediated RF response under a prescribed NLoS interaction. Finally, a separate room-scale campaign evaluates the resulting prior using exhaustive TX-RX beam maps without prescribing the underlying propagation mechanism. The measurements show that descriptor reliability depends on acquisition geometry and point-cloud representation, and that LiDAR and RF surface responses exhibit cross-modal association without supporting deterministic RF-power prediction. In the room experiment, local three-ring 3-D planarity retains a beam within 3 dB of exhaustive search at 74.5% of the measured locations while reducing RF beam-pair probing by 72% relative to exhaustive probing over the candidate search region. These results establish LiDAR-derived local surface structure as a communication-oriented prior for concentrating RF probing and reducing mmWave beam-search uncertainty.

0.20

TriSAR-3D and PaintFusion: A Novel Trimodal LiDAR-RGB-Thermal Dataset and Method for 3D Person Detection in Search and Rescue

cs.CV cs.RO ·2026-10-15 ·Markus Kramer, Jan Wachenfeld, Oskar von Stryk
code/data similarity 0.61
Abstract

Reliable 3D person detection is important for rescue robotics to give first responders situational awareness. Yet most multimodal 3D detection research targets automotive scenarios in which pedestrians mostly appear in a narrow range of upright, near-road poses. It is therefore unclear how much of the reported performance transfers to rescue settings, as a person may be lying, partially occluded, or in unstructured indoor and outdoor environments. To investigate this gap, we introduce TriSAR-3D, a multimodal person detection dataset recorded with a high-resolution 128-channel LiDAR, an RGB camera, and a thermal camera. It captures varied poses in indoor as well as outdoor locations at different times of day to study their influence on 3D object detector performance. Furthermore, we propose a novel trimodal 3D detection pipeline, PaintFusion, that fuses PointPainting-based evidence using a simple MLP. We show that the derived method reaches 65.4 AP@50 and thereby outperforms strong LiDAR-only (CenterPoint, 53.4), RGB+LiDAR (BEVFusion, 60.5), and adapted trimodal (Tri-BEVFusion, 60.7) baselines on the dataset. At the same time, it is substantially faster than other fusion-based methods such as BEVFusion, with a per-frame latency of 51.6 ms compared to 101 ms. Additionally, we test how the performance of different 3D object detectors generalizes to unseen pose categories. Removing lying instances from training data collapses the lying-pose AP@50 to near zero across all evaluated detectors. The results indicate that dataset diversity exerts a large effect on 3D object detection performance. The dataset and code are available at https://github.com/MarkusKramer1/3D-Trimodal-People-Detection.

0.23

TriO: Tri-Modal Unsupervised Occupancy World Model for Anything Perception

cs.CV cs.AI ·2026-09-25 ·Quinlan Sykora, Sourav Biswas, Christopher Diehl, Andrew Cunningham, Thomas Gilles +1 more
top venue similarity 0.60
Abstract · Published at ECCV 2026, 49 pages, 20 figures

We present TriO, a multi-modal unsupervised world model that predicts 4D occupancy, obstacle segmentation, flow and LiDAR. In contrast to prior work, TriO utilizes three distinct sensor modalities (camera, LiDAR, and RADAR) as both inputs and sources of self-supervision, eliminating the need for additional human annotations. Thanks to its novel supervision, the model is able to segment any occupancy from the drivable surface, overcoming the limitations of existing open-set methods in handling long-tail objects. TriO achieves state-of-the-art results in multiple 3D and 4D tasks, including occupancy, flow, and LiDAR prediction, as well as zero-shot road obstacle segmentation across multiple datasets such as Argoverse 2, and Spotting the Unexpected.

0.40

GNSS Anomaly Detection in Urban Canyons: A Comprehensive Survey of Multimodal Visibility Inference

eess.SP ·2026-10-25 ·Abdelgabar Ahmed, Tarig Ballal, Nour Kouzayha, Tareq Y. Al-Naffouri
publishedsurvey similarity 0.59
Abstract · Accepted for publication in IEEE Aerospace and Electronic Systems Magazine. DOI: 10.1109/MAES.2026.3746915

Reliable GNSS positioning in dense urban environments depends on assessing satellite visibility and identifying measurements degraded by non-line-of-sight (NLOS) reception and multipath. This survey organizes reception-state inference into three complementary evidence families: receiver-side signals, 3D maps and environmental models, and auxiliary sensors. Receiver-side approaches span antenna/electromagnetic, tracking/baseband, and observable-domain evidence, interpreted through rule-based, statistical, machine-learning, and deep-learning methods. Map-based approaches are examined by reviewing environmental representations and methods such as geometric screening, shadow matching, 3D ray tracing, and neural implicit visibility prediction. IMU, camera, and LiDAR measurements provide complementary kinematic, visual, and geometric evidence for online screening. Across these families, we compare accuracy, interpretability, hardware and calibration requirements, map dependence, computational cost, and suitability for real-time deployment. We identify open challenges in dataset quality and labeling, domain generalization, map freshness, and resource-efficient implementation. The resulting synthesis connects signal processing, environmental modeling, and sensor fusion to guide the design of robust urban GNSS measurement-screening systems.

0.26