CS Paper Radar

VISTA: A Framework for Vision-Informed Semantic Traffic Awareness

Murat Arda Onsu, Poonam Lohan, Burak Kantarci, Aisha Syed, Matthew Andrews, Sean Kennedy

cs.CV cs.LGcs.NI ·submitted 2026-11-16 03:26 UTC · v1
published

Abstract

The Internet of Things (IoT) is transforming intelligent transportation systems by connecting roadside cameras, edge computing units, vehicles, and traffic infrastructure into a shared sensing and decision-making ecosystem. However, transmitting raw traffic video over vehicle-to-everything (V2X) links is bandwidth-intensive, while deploying large multimodal models directly on resource-constrained vehicles is often impractical. To address these limitations, this article proposes VISTA (Vision-Informed Semantic Traffic Awareness), a multimodal vision-language edge-intelligence framework that combines predictive video representation learning with lightweight language-based reasoning to provide communication-efficient traffic situational awareness. In the proposed architecture, roadside surveillance units employ the Video Joint Embedding Predictive Architecture (V-JEPA) and an attentive-pooler module to convert traffic video clips into compact future-scene embeddings. These embeddings are transmitted through V2X links to vehicle-mounted systems, where a TinyLlama small language model, enhanced with a custom visual adapter, interprets the embeddings alongside natural-language (NL) traffic queries. VISTA enables vehicles to obtain interpretable traffic information, reason about upcoming scene dynamics, and identify potential collision risks without receiving raw video streams. Experimental results in a digital-twin urban traffic environment show that VISTA achieves 97.77% accuracy and an 89% F1 score for future collision prediction while reducing the communication payload by several orders of magnitude compared with raw RGB video transmission. These results demonstrate the feasibility of embedding-based multimodal intelligence for low-latency, communication-efficient, and safety-oriented IoT-enabled transportation systems.

Comment
8 pages, 4 figures, Accepted for publication in IEEE Internet of Things Magazine, 2026

Closest papers

Nearest neighbours by meaning across the whole index.

VLALight: A Vision-Language-Action Model for Traffic Signal Control

cs.AI ·2026-09-29 ·Pan Zhang, Siqi Lai, Kemu Dong, Hao Liu
code/data similarity 0.72
Abstract

Traffic signal control (TSC) is essential for improving urban mobility and reducing congestion. Although roadside cameras are widely deployed at signalized intersections and provide rich visual observations of evolving traffic, existing TSC methods typically rely on manually engineered traffic states or separate perception modules, creating a gap between physical observations and control decisions. We present VLALight, the first vision-language-action (VLA) model for end-to-end traffic signal control from multi-view roadside videos. VLALight directly maps visual observations to coordinated signal actions through multi-target spatiotemporal traffic reasoning and topology-aware cooperative perception across intersections. To establish this capability, we develop a two-stage supervised cold-start training strategy for visual traffic understanding and signal decision-making, followed by cooperative agentic reinforcement learning that jointly optimizes local control and network-wide traffic efficiency. Furthermore, VLALight introduces adaptive fast and slow reasoning modes, enabling the policy to allocate deeper reasoning only when additional deliberation provides sufficient control benefits. Through balanced mode-aware rollouts and relative advantage optimization, VLALight learns to trade off decision quality and inference cost. Extensive experiments on seven real-world traffic-flow datasets across three urban networks demonstrate that VLALight consistently outperforms transportation-based, RL-based, and LLM/VLM-based baselines. Ablation studies validate the effectiveness of cooperative perception, network-level optimization, and adaptive reasoning. These results demonstrate the potential of VLA models for real-world physical traffic control. Our project is available at https://github.com/usail-hkust/VLALight.git.

0.23

Traffic-VL: Reinforced Vision-Language Models for Generalizable End-to-End Traffic Signal Control

cs.CV cs.AI ·2026-11-04 ·Xingchen Zou, Maonan Wang, Kemou Jiang, Jie Li, Yiqi Chen +3 more
similarity 0.71
Abstract

Existing traffic signal control methods typically rely on structured traffic states produced by dedicated perception pipelines. Emerging visual approaches often use idealized simulator images and convert them into intermediate representations before control, limiting their ability to directly exploit roadside visual observations. To bridge both the visual-domain and perception-to-decision gaps, we introduce SynTra-V and Traffic-VL. SynTra-V converts physics-based traffic simulations into photorealistic multi-view surveillance imagery across diverse road topologies and sensing conditions, and automatic semantic auditing checks that control-critical geometry, vehicle movements, and lane semantics match the simulator state. Traffic-VL is an eight-billion-parameter vision-language model that directly maps synchronized multi-view images and natural-language phase descriptions to signal decisions without intermediate structured states. Building on Traffic-R1's two-stage reinforced reasoning framework, Traffic-VL adapts its vision encoder via offline reinforcement learning on SynTra-V and freezes it during online visual stepwise trajectory policy optimization in closed-loop simulation. Experiments show that Traffic-VL matches structured-state Traffic-R1 while outperforming image-based controllers and general-purpose vision-language models, including Gemini 3 Pro. Evaluations on unseen topologies, degraded sensing conditions, real surveillance footage, and a three-week deployment across ten intersections further demonstrate robust visual generalization and effective constrained online control.

0.10

VLALight: Lightweight Vision-Language-Action Models for Emergency-Aware Traffic Signal Control

cs.CV cs.AI ·2026-09-25 ·Kemou Jiang, Maonan Wang, Xingchen Zou, Jiayue Zhu, Yuhang Fu +4 more
similarity 0.70
Abstract · 9 pages, 7 figures

Traffic signal control (TSC) is essential for mitigating urban congestion. Recent advances in vision-language models (VLMs) enable richer interpretation of intersection scenes, opening new opportunities for visual-context-aware TSC. However, the loose coupling and repeated information conversion between modules can lead to the loss of fine-grained visual details, while sequential inference introduces substantial latency. To address these limitations, we propose VLALight, a lightweight end-to-end vision-language-action framework that directly maps intersection observations and signal-phase information to discrete signal actions. To handle the multi-view nature of TSC, VLALight combines multiple directional camera views into a unified visual input and uses textual instructions to establish their correspondence with traffic movements and signal phases. This design enables direct action prediction with a compact 0.5 B-parameter model, without intermediate image-to-text descriptions or handcrafted traffic-state representations. Experiments show that VLALight delivers the best emergency-vehicle service of all compared methods, reducing pooled emergency waiting time by 21.1% over the cascaded VLMLight while running in real time on local hardware and generalizing to unseen intersection topologies and traffic-flow patterns.

0.22

Sim-to-Real Traffic Scene Understanding by Decoupling Semantics from Caption Generation with V-JEPA

cs.CV ·2026-09-16 ·Nguyen Hoai Thuong Bui, Thanh Nguyen Vo, Trinh Tra Giang Nguyen, Ha Duc Bui
top venuebenchmark similarity 0.69
Abstract · Winner of Track 2 at the AI City Challenge 2026, with the paper published at the ECCV conference 2026 (ECCV-W)

Track 2 of the AI City Challenge 2026 requires both visual question answering (VQA) and traffic event description generation under a challenging synthetic-to real domain shift. Existing vision-language approaches often entangle semantic understanding with language generation, making them susceptible to hallucination and inconsistent reasoning across event phases. In this work, we propose a decoupled semantic understanding framework that first resolves predefined traffic questions into structured semantic facts and subsequently uses these facts to guide caption generation. A frozen V-JEPA encoder extracts predictive scene representations, while a lightweight Llama-based predictor produces answers for VQA queries. To improve reliability, we introduce a training-free structured refinement mechanism that exploits statistical priors, inter-question relationships, and temporal event consistency to correct prediction errors. The refined semantic facts are then provided to Qwen3-VL-8B to generate pedestrian and vehicle descriptions for each traffic event. Experimental results on the official 2026 AI City Challenge Track 2 benchmark show that the proposed method achieves 87.09% VQA accuracy and an overall S2 score of 60.0853, ranking first among all participating teams. These results demonstrate that predictive world representations combined with structured semantic refinement enable more accurate and reliable traffic understanding, leading to higher-quality lan guage generation.

0.59

Grounding Vision-Language Models in Driving Semantics: A Multi-Dataset Predicate Framework for Explainable Reasoning

cs.LG cs.CVcs.RO ·2026-09-10 ·Mohamed Chouai, Fazli Faruk Okumus, Stefan Kugele
similarity 0.69
Abstract

Vision-language models are increasingly used for driving-scene understanding, yet the semantic relations expressed in their outputs are often difficult to verify against the underlying traffic situation. This paper introduces a deterministic multi-dataset predicate framework that derives driving-scene semantics from measurable geometric, kinematic, temporal, map, and traffic-control evidence. Dataset-specific interfaces are used only to recover the required scene information, while predicate definitions remain unchanged across nuPlan and nuScenes and are materialised in a common Predicate Knowledge Graph. Quantitative semantic validation against manually annotated predicate relations on 200 scenarios from each dataset yields macro F1 scores of 0.94 on nuPlan and 0.93 on nuScenes, with an average cross-dataset difference of 0.02 across the shared predicates. The Predicate KG is further evaluated using a frozen LLaVA-OneVision-7B model on the nine NuPlanQA subtasks. Predicate grounding achieves the highest accuracy among the evaluated visual-input conditions in seven of nine NuPlanQA subtasks, including Traffic Light (53.2% to 71.5%), Situation Assessment (76.2% to 86.1%), and Action Recommendation (82.9% to 89.0%). Weather/Lighting remains essentially unchanged (89.4% vs. 88.8%), consistent with the absence of corresponding predicates, while Predicate KG only input outperforms metadata-only input in eight of nine subtasks. The results show that deterministic predicates provide a consistent and traceable semantic representation and, under oracle grounding, can reduce visual dependence for reasoning tasks covered by the predicate vocabulary.

0.10

Roadside-Cooperative Autonomous Driving: From Data Platform to Vision-Language End-to-End Reasoning

cs.RO ·2026-08-21 ·Yitao Xu, Tong Wu, Yiyan Wu, Guoji Xu, Yanbo Jiang +5 more
benchmark similarity 0.68
Abstract · 15 pages, 5 figures, under review

Vehicle-to-Everything (V2X) cooperation enables beyond-line-of-sight perception, mitigating occlusions in single-vehicle sensing. However, existing V2X benchmarks provide limited support for closed-loop evaluation and language-grounded supervision, hindering the development of vision-language models (VLMs) for end-to-end cooperative driving. To address these limitations, we introduce V2XBench, a simulation platform featuring synchronized ego--roadside sensing and closed-loop evaluation, together with Chat-V2XBench, a progressively structured VQA dataset for cooperative reasoning. Building upon this benchmark infrastructure, we propose AURORA, an end-to-end cooperative driving framework. Equipped with a dual-view perception architecture, AURORA mitigates spatial and semantic discrepancies across ego and roadside viewpoints through a query-level Cross-View Query Alignment and Fusion (CQAF) module. Leveraging the resulting unified tokens, a LoRA-adapted VLM bridges semantic reasoning and generative trajectory planning. Extensive closed-loop evaluations on V2XBench demonstrate that AURORA achieves state-of-the-art performance in heavily occluded scenarios, with a Route Completion rate of 98.21% and a Driving Score of 76.02, while requiring low roadside communication bandwidth. Ultimately, this work pioneers an extensible V2X--VLM paradigm, paving the way for next-generation cooperative autonomous driving.

0.32

Beyond Vehicle-Centric Data: Scaling Autonomous Driving through Roadside-to-Vehicle Generation

cs.CV ·2026-11-15 ·Yichen Zhang, Zihan Wang, Hao Chen, Runyi Huang, Ruixue Zong +4 more
dataset similarity 0.67
Abstract

Training autonomous driving models requires large and diverse datasets, collected primarily by vehicles. Scaling them entails investment in fleet operation, data processing and annotation. Road infrastructure records traffic and multi-vehicle interactions, yet the viewpoint gap between roadside and onboard sensors limits its direct use for training. Here we introduce AnyR2V, a framework that transforms multimodal roadside observations into seven-camera surround-view videos for different vehicle perspectives. It aggregates sequential observations into a shared scene representation of geometry and road-user motion, then renders appearance, depth and layout conditions along target-vehicle trajectories. Supervised by spatiotemporally aligned roadside-vehicle pairs, AnyR2V generates surround-view videos guided by scene structure and road-user motion. Existing roadside 3D bounding boxes are transformed into target-vehicle coordinates and paired with generated images, allowing one traffic recording and its annotations to yield training data from multiple vehicle perspectives. We construct R2V360, a dataset of 89 real traffic sequences at an urban intersection, with aligned multimodal roadside data and onboard surround-view videos. AnyR2V outperforms six scene reconstruction methods in image fidelity and in depth and semantic agreement between generated and real onboard images. We further evaluate generated videos along other vehicles' trajectories by adding them to real training data. Adding generated frames equal to 50% of the original real training frame count improves mean average precision (mAP) by 3.3-3.4 AP-points across three 3D object detectors on the real onboard test set, relative to real-data-only baselines. These results demonstrate that roadside traffic records can provide training data from multiple vehicle perspectives, offering a new data source for autonomous driving.

0.15

Semantic Edge-Cloud Communication for Real-Time Urban Traffic Digital Twins

cs.NI cs.CVcs.DCeess.SP ·2026-10-09 ·Murat Arda Onsu, Arman Dogru, Poonam Lohan, Burak Kantarci, Aisha Syed
published similarity 0.67
Abstract · 6 pages, 4 figures, accepted to IEEE Global Communications (Globecom) Conference, 2026

Urban traffic systems are becoming increasingly complex, demanding real-time traffic perception, proactive collision prediction, and intelligent management to improve safety and reduce congestion. Although AI-driven traffic management has shown strong promise, its success relies on diverse, high-quality data spanning critical scenarios such as collisions, near-collisions, and traffic rule violations. Digital twins offer a powerful solution by enabling synchronized virtual replicas of physical traffic environments for real-time monitoring, collision prediction, traffic-flow optimization, and diverse realistic scenario generation. However, continuously synchronizing a high-fidelity urban traffic digital twin through edge surveillance cameras requires substantial edge-cloud communication, and transmitting raw video streams leads to severe bandwidth and latency constraints. To overcome this challenge, this paper proposes a semantic communication-based edge-cloud framework for real-time urban traffic digital twin applications. By leveraging edge intelligence to extract and transmit only task-relevant semantic traffic information instead of full video feeds, the proposed framework significantly reduces communication overhead while preserving the essential dynamics needed for accurate real-time digital twin reconstruction. Experimental results demonstrate more than 99% transmission savings and an average trajectory reconstruction error of only 0.12 m, confirming the proposed framework as an effective, scalable, and low-latency solution for AI-enabled urban traffic digital twin applications.

0.43