CS Paper Radar

The Calibration Illusion in Traffic Microsimulation

Cameron Hickert, Maryam Samaei, Athena Wang, Chengyuan Zhang, Lijun Sun, Yanbing Wang, Mostafa Ameli, Cathy Wu

eess.SY ·submitted 2026-08-20 05:17 UTC · v1
benchmark

Abstract

The transportation community seeks to use calibration methods for highway traffic microsimulation. This is a response to the time-consuming and subjective nature of traditional manual calibration, as well as the growing prevalence of data for calibration. This work argues that this "automatic" calibration is largely an illusion. A significant - and unquantified - amount of bespoke manual work is hidden behind these methods. This illusion inhibits a core component of scientific advancement: objective comparison against a shared standard. This impedes evaluation, undermines reproducibility, and fragments research. To address this gap, this paper introduces a comprehensive benchmark designed to simultaneously expose the calibration illusion for highway microsimulation and provide a common ruler. The results across a range of scenarios present a new baseline for what the algorithms can achieve without bespoke tuning, revealing the research gap that remains and providing a tool to advance a cumulative science of calibration. Additional experiments provide insights into the source of calibration errors that arise in large-scale highway calibration relative to the simplified settings under which methods are commonly developed.

Comment
Working Paper, 41 pages, 11 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Beyond corridor averages: Lane-level validation of microscopic freeway simulation with empirically parameterized arrivals

cs.CE ·2026-11-18 ·Korede R. Bishi, Casey Bowman, John A. Miller
similarity 0.59
Abstract

Traffic flow modeling is an essential part of civil planning around the world, and traffic simulation is an important component of the analyses that city planners must perform to ensure safe and efficient road networks for their citizens. This work advances microscopic traffic simulation through lane-level validation of speed and flow dynamics, addressing a critical gap in conventional macro-aggregated approaches. We systematically evaluate the trade-offs between numerical integration methods and vehicle arrival processes when reproducing empirical trajectory data from California's freeway network. Results demonstrate that numerical-integrator choice has negligible impact on simulation accuracy, with less than one percent variation across eight methods, while vehicle arrival-process modeling substantially affects fidelity. The shifted Erlang-2 distribution reduces flow-prediction error by approximately 28% compared to Poisson processes by enforcing a realistic minimum headway. Lane-level validation across five detector stations reveals lane-specific dynamics that aggregated metrics often obscure.

0.49

On the Input Sensitivity of METANET Models and the Robustness of Dynamic Calibration

eess.SY ·2026-09-30 ·Cameron Hickert, Shreyaa Raghavan, Cathy Wu
published similarity 0.57
Abstract · Accepted at the 65th IEEE Conference on Decision and Control. C. Hickert and S. Raghavan contributed equally to this work. 8 pages, 5 figure…

While traffic modeling and control rely upon effective calibration of macroscopic traffic simulation models like METANET, recent empirical work shows these models can exhibit severe sensitivity to input noise. This paper explains this phenomenon through a string-stability analysis, demonstrating that a calibrated METANET model can amplify small additive perturbations to boundary conditions along the corridor, causing the simulated state to diverge from the nominal baseline. The input sensitivity in off-nominal cases may compromise the model's capabilities for counterfactual analysis, which is required for use cases of interest like the design of large-scale variable speed limits. To address these issues, this work demonstrates how a dynamic, time-varying approach to calibrate model parameters can achieve robustness and improved accuracy of the resulting macrosimulation. We show analytically that under stated regularity and perturbation assumptions, dynamic calibration achieves a tighter cost deviation bound than static calibration, and we validate this behavior in highway environments with synthetic and real-world data. Ultimately, this can enable more trustworthy traffic simulation and more effective evaluation of control strategies.

0.26

Empirically Calibrated Differential Games for Simulating Driver Interactions

cs.MA math.OC ·2026-10-23 ·Amirmohammad Khakpour, Nachuan Li, Pedram Beigi, Yu, Nie
published similarity 0.56
Abstract · This manuscript is intended for submission to IEEE Transactions on Intelligent Transportation Systems (IEEE T-ITS)

Game-theoretic models can resolve vehicle interactions without hand-written priority rules, but whether they describe human drivers once calibrated has rarely been tested. This study formulates a general-sum differential game of vehicle interactions at urban intersections, calibrates it to observed driving, and evaluates it on data not used for calibration. Each vehicle minimizes a cost that combines speed tracking, adherence to a map-derived lane route, control effort, gap keeping based on the intelligent driver model, a predictive conflict term, and a contact barrier, while perceiving the other vehicles with a delay. The game is solved with a feedback iterative linear-quadratic algorithm, and ten behavioral parameters per vehicle are calibrated by Bayesian optimization on trajectories recorded at four intersections in Washington, D.C. The calibrated game reproduces the trajectories of 819 vehicles with a mean position error of 1.00 m. When only each vehicle's route is known, as in a simulator, it is the most accurate of the compared models on 96 held-out vehicles (7.42 m), and ten-fold cross-validation shows that it is significantly more accurate than calibrated SUMO (by 3.61 m), an inverse reinforcement learning model (by 0.93 m), and independent and cooperative variants of the same model (by 0.33 and 0.32 m). The solutions, computed in 2.1 s for a typical interaction, are verified approximate equilibria. Finally, the conflict and steering weights, which the data do not identify, can be set from safety and comfort targets; together with a stiffer contact barrier, this removes the collisions that otherwise occur in 43% of tight closed-loop conflicts.

0.34

Classifying Urban Locations by Cyclist Facility Usage and Directional Alignment: A Validated Camera-Based Framework from Multi-Site Observations

cs.CV ·2026-10-27 ·Ao He, Mohit Gupta, Deborah Salon, Ricardo Eiris, Vivian W. H. Wong
publishedbenchmarkunusual similarity 0.56
Abstract · 19 pages, 6 figures, 2 tables. Accepted for presentation at the 106th Transportation Research Board (TRB) Annual Meeting, Washington, D.C., …

Objectives: Agencies invest in bicycle facilities but lack tools to assess whether cyclists use them as designed. We examine whether usage patterns follow facility configurations and whether higher bike lane usage is associated with less wrong-way riding. Methods: Motion-triggered cameras covered 19 deployments at 17 locations in Tempe, Arizona, in 2023. Manual review identified 1,963 cycling events; excluding 20 crossings left 1,943 labeled events for analysis. An automated pipeline on the same captures counted 92.9 percent of the full manual total (per-deployment r = 0.99), and every automated wrong-way call was visually verified. Deployments were classified using thresholds on facility usage shares. Findings: Deployments were bike-lane dominant (9), sidewalk dominant (6), or mixed (4). Cyclists in bike lanes rode against the flow in 4.6 percent of events; on sidewalks, riders opposed the near-side traffic flow in 49.6 percent. Across deployments, bike lane share and wrong-way rate were negatively correlated (Spearman rho = -0.87, p < 0.001, n = 16). An independent volunteer count in the same city showed a similar sidewalk-direction pattern (rho = 0.79). Novelty: The study combines multi-site still-camera observations with a fully hand-verified benchmark to characterize facility use and travel direction. Practical Applications: The workflow offers planners a low-cost screen for locations with high sidewalk cycling and travel against the near-side traffic flow, supporting more detailed site investigation.

0.39

TrafficSignBench: Rule-Centric Closed-Loop Evaluation of Traffic-Sign Compliance in Autonomous Driving

cs.RO cs.CV ·2026-09-29 ·Victoria Smirnova, Viktoriia Zinkovich, Gregorii Bukhtuev, Artem Belyaev, Andrey Kuznetsov +2 more
benchmark similarity 0.56
Abstract

Autonomous driving planners are typically evaluated using aggregate metrics such as driving score, destination rate, and collision rate, which do not explicitly measure compliance with traffic rules. As a result, planners can achieve high benchmark scores while still exhibiting unsafe or illegal behaviors, limiting their applicability to real-world deployment. To address this gap, we introduce TrafficSignBench, a large-scale, traffic sign-centric benchmark for systematic and interpretable evaluation of traffic-rule compliance in autonomous driving. Our framework combines real-map-based simulation for realistic road layouts with rule-targeted procedural scenario generation for scalable and balanced coverage of underrepresented rules. We implement traffic rules corresponding to 34 traffic signs, each equipped with an automatic rule checker for detecting violations during closed-loop execution. This design yields 29,000 diverse road scenes and 29 distinct testing scenario types, enabling controlled evaluation of rule-specific planner behavior. We construct 5,800 testing scenes and demonstrate that current autonomous driving planners can exhibit poor traffic-rule compliance despite strong performance on standard evaluation metrics. To address this limitation, we transform existing planners into rule-compliant trajectory experts via explicit traffic-sign constraints, enabling scalable generation of high-quality oracle trajectories for fine-tuning.

0.17

Assessing the Impact of Fleet Size on Crowdsourced Mapping Using a Dissimilarity Measure

cs.RO physics.data-an ·2026-09-24 ·Marie-Ngoïe Badibanga Kalenda, Philippe Bonnifait, Marie-Anne Mittet
published similarity 0.56
Abstract

Accurate digital maps are essential for Advanced Driver Assistance Systems (ADAS) or Autonomous Driving (AD), providing critical information such as road geometry, traffic signs and speed limits required by safety functions including Intelligent Speed Assistance (ISA). Maintaining these map layers using traditional surveying methods is costly and difficult to scale. Crowdsourced approaches based on fleets provide a promising alternative for continuously validating and updating map information. However, the relationship between the number of contributing vehicles and the quality of the resulting map remains poorly understood. To address this gap, this paper presents a simulation-based framework for evaluating crowdsourced traffic sign maintenance using a dissimilarity measure called GOSPAM (Generalized Optimal SubPattern Assignment for Maps), which combines localization errors with detection performance by accounting for False Positives (FP) and False Negatives (FN). The proposed system models multivehicle observations with representative sensor noise, detection errors, and semantic recognition uncertainties. Observations from multiple vehicles are aggregated using spatial clustering and semantic filtering to estimate traffic sign locations. Using simulated trajectories generated from data carried out by an experimental vehicle in an area containing ground-truth traffic signs, we assess the influence of fleet size on the performance of crowdsourced mapping. The number of vehicles ranges from 5 to 50, and performance is analyzed using standard evaluation metrics which are compared to the GOSPAM . The results show that GOSPAM can be used to effectively assess the quality of crowdsourced mapping, such as the contributions made by the first vehicles or the improvements made by numerous vehicles.

0.38

The Reality Gap in Automated Driving Systems Testing: Landscape, Challenges, and Research Directions

cs.SE cs.RO ·2026-10-28 ·Stefano Carlo Lambertenghi, Oliver Weissl, Johannes Betz, Andrea Stocco
similarity 0.55
Abstract

Simulation is essential for testing automated driving systems (ADS), but discrepancies between simulation and reality can undermine testing conclusions. It remains unclear which discrepancies the literature addresses and what evidence supports correspondence claims. We present a systematic mapping of 235 primary studies that measure or mitigate the reality gap in ADS testing, selected from 4,195 publications. We characterise objectives, targets, approaches, experimental setups, and evaluation evidence. Our classification distinguishes simulated and real-world execution across six ADS test components, separately for each approach and its evaluation reference. This distinguishes real-world information used to construct an approach from evidence used to evaluate its claims. Of the included studies, 76% address mitigation, primarily targeting scenario and actor realism and perception, while 94% report validation or measurement procedures. Only 61 studies report metrics from multiple families covering sensor/perception correspondence, ADS behaviour, vehicle dynamics, or safety outcomes; this coverage does not establish causal links between discrepancies and test verdicts. The synthesis highlights that real-world content and physical execution must be interpreted against the claim evaluated. The findings motivate claim-specific evaluation and component-level reporting to establish which discrepancies affect testing conclusions and when simulation evidence supports conclusions about real-world ADS behaviour.

0.08

Ego2Flow: Benchmarking the Traffic-System Impact of Autonomous Driving on Maps Distilled from Real-World Roads

cs.RO cs.CV ·2026-11-13 ·Yu Dong, Weiying Shen, Hangyu Li, Shu Han, Hong Li +6 more
benchmarkcode/data similarity 0.55
Abstract

End-to-end autonomous driving is predominantly evaluated through ego-vehicle safety and task completion. Its deployment in urban traffic, however, creates sustained interactions among vehicles whose collective effects remain insufficiently evaluated. We introduce Ego2Flow, a closed-loop benchmark for assessing the traffic-system impact of autonomous driving on maps distilled from real-world cities. To construct realistic 3D urban environments for sustained closed-loop traffic simulation, we develop an agent-guided map distillation framework that integrates real-world maps and observations with complementary geographic information. Through an iterative feedback mechanism and a traffic information graph that jointly guide scene generation, the framework preserves lane topology, critical road geometry and decision-relevant visual conditions while simplifying nonessential details and completing unobserved regions. Using our proposed framework, we generate and publicly release six CARLA maps from the Argoverse 2 dataset, covering the corresponding real-world regions and enabling paired comparisons of driving behavior under real and simulated observations. On these maps, Ego2Flow evaluates how autonomous driving policies affect traffic efficiency, safety, stability and energy consumption across diverse road structures. Compared with existing construction methods, our framework achieves the lowest average displacement error between planning outputs from paired real and simulated observations. We further benchmark representative driving policies to characterize their system-level impact on urban traffic. Project Page: https://github.com/Fisher-007/Ego2Flow.

0.28