CS Paper Radar

Measuring Decision-Scale Use in Tool-Augmented LLMs: A Contrastive Urban Benchmark

Ray Chen, Vivian Wong, Christan Grant

cs.IR ·submitted 2026-09-15 04:06 UTC · v2
benchmark

Abstract

Urban decision-support often asks whether activity is unusually high or low for a specific place, not which place has the larger raw count. Twenty pickups in a quiet neighborhood can be more abnormal than 180 at an airport. We introduce URBANCONTRASTIVEQA, a benchmark that asks whether tool-augmented language models can make this baseline-relative comparison. Each item pairs two urban situations from public mobility data in NYC, Chicago, and Seattle, labeled by how far current activity deviates from that place's historical baseline. We evaluate six instruction-tuned models under five tool-output formats. With only raw counts, models often pick the larger number even when it is less abnormal for its zone. Server-computed baseline scores and ordinal labels raise accuracy, but gains vary by model. For heterogeneous urban feeds, tool interfaces need to expose local baselines, not just activity volumes. We release the pair bank, labels, scoring scripts, and data card.

Closest papers

Nearest neighbours by meaning across the whole index.

Do LLMs Know Your Neighborhood? Auditing LLM Priors for Neighborhood-Level Mobility Prediction and Structural Alignment

cs.LG cs.CY ·2026-08-31 ·Saad Mohammad Abrar, Eesha Kurella, Arnav Dadarya, Naman Awasthi, Kazi Tasnim Zinat +1 more
similarity 0.62
Abstract

Human mobility is central to urban planning, transportation, public health, and emergency response, yet fine-grained trajectory data are often proprietary, restricted, and privacy-sensitive. Large language models (LLMs) offer a potential alternative by generating plausible mobility traces and predicting individual movement, but their ability to infer aggregate neighborhood-level mobility remains unclear. We evaluate zero-shot LLMs on Census Block Group-level mobility prediction across four U.S. metropolitan areas using anonymized Cuebiq data to construct point-level, trajectory-level, and temporal mobility outcomes, paired with sociodemographic and built-environment predictors. We compare LLM predictions with supervised baselines and introduce a directional alignment analysis to test whether LLM-implied predictor effects agree with empirical OLS and Jonckheere-Terpstra trends. Supervised models achieve 0.580 average accuracy, compared with 0.435 for the best LLM, with spatial extent outcomes showing the strongest predictability but also the largest LLM-baseline gaps. Directional analysis shows that LLMs often rely on coarse, stable predictor-level priors that remain similar across outcomes and cities, including asymmetric treatment of protected-group predictors. Overall, LLMs can partially recover aggregate mobility patterns from urban context, but their predictions should not be treated as structurally grounded without auditing empirical alignment and potential bias.

0.18

SeoulMMOD: A Large-Scale Multimodal Origin-Destination Flow Benchmark

cs.LG ·2026-10-09 ·Taeyoung Yu, Seonbin Jo, Jiwon Kim, Junyoung Byun
benchmark similarity 0.61
Abstract

Origin-destination (OD) flow forecasting supports urban mobility decisions by predicting the volume and patterns of human movement across a city. Most existing public OD benchmarks provide only a partial view of citywide mobility, covering a single transport service or at most two modes. With partial mode coverage, observed demand changes are difficult to separate into shifts across transport modes and changes in total travel. We introduce SeoulMMOD, a large-scale public benchmark for citywide multimodal mobility in Seoul. SeoulMMOD provides three years of hourly OD flows estimating total mobility for 6 urban travel modes across 25 districts and 426 sub-districts, together with travel-time, travel-distance, calendar, rainfall, and point-of-interest (POI) signals. Using SeoulMMOD, we benchmark 14 forecasting baselines under a common protocol and analyse the effects of spatial scaling, joint versus mode-separate training, and cross-year generalization. Results show that several spatio-temporal models that work at district level do not scale to sub-district OD forecasting, and that naive joint multi-mode training does not consistently improve accuracy. Together, the dataset and benchmark establish a reproducible testbed for multi-mode, multi-year OD forecasting at the city scale.

0.04

EdgeNet City: Context-Aware AI-RAN Planning toward Autonomous Urban Connectivity

cs.NI ·2026-11-14 ·Ziming Liu, Boxuan Shan, Qixing Deng, Wenchao Xue, Xiaoli Chu
similarity 0.60
Abstract · 6 pages. Submitted to IEEE ICC 2027 (Mobile and Wireless Networks). Author manuscript

The evolution from 5G toward 6G calls for more anticipatory and context-aware Cell-on-Wheels (COW) planning in wireless networks. We present EdgeNet City, an online framework that grounds free-form natural-language inputs in geographic activity representations and path-gain data. A catalog-referenced representation links activity roles, geographic features and predefined demand profiles, separating semantic selection from deterministic spatial computation. The algorithm combines prescribed demand with relative propagation support to screen service areas for subsequent COW assessment. We publicly release a London dataset with 22,678 transmitters, approximately 449.4 billion path-gain records and retained receiver geometry. Within prepared propagation domains, the online prototype reuses these records without new ray tracing or live operator measurements. In a matched 20-event study across six London environments, the expanded representation increases mean spatial overlap with the reused full technical-review reference from 0.4929 for the single-feature Venue-only baseline to 0.7203. Rule-based and language-model planners select identical activities in this panel. With need-aware ranking and a three-cell diagnostic budget fixed, the expanded representation increases mean Reference-need Capture by 25.44 percentage points over Venue-only.

0.17

MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes

cs.LG cs.AIcs.CL ·2026-09-09 ·Remco Hendriks
benchmarkcode/data similarity 0.59
Abstract · 23 pages, 5 figures, 10 tables. Code and data at https://github.com/continker/metrollm-bench (tag paper-v1.2); DOI 10.5281/zenodo.21893944

We introduce MetroLLM-Bench, a 955-case benchmark for testing language models as the policy layer of a transit kiosk. It covers six real metro systems, ranging from 37 to 414 stations, and eleven categories that include routing, fare calculation, disruptions, accessibility, and adversarial input. In each case, the model must call structured tools and submit a machine-renderable terminal state containing an outcome, a per-ticket fare quote when applicable, and a kiosk action. Fourteen deterministic scoring components form Tier 1; eight semantic-quality components form Tier 2, six of which use a language-model judge. We report Tier 1 and the combined score of both tiers. A stratified 75/25 split reserves 717 cases for training-data generation and 238 for held-out evaluation. We evaluate twenty-six models from six vendors, of which twenty-three are ranked. On the held-out partition, a 4B Qwen 3.5 student trained through parameter-efficient fine-tuning (PEFT) exceeds both GPT-5.6 tiers on Tier 1 (91.3 against 90.6 and 90.0) and matches GPT-5.4 full at maximum reasoning effort (91.4), with a 2.6 GB Q4_K_M footprint. Larger 9B and 27B students provide no further Tier 1 improvement over the 4B student at this training scale. Across the four Qwen sizes, the PEFT gain over the corresponding base model decreases from +7.03 points at 2B (three training seeds) to -0.91 at 27B; every seed shows the same direction at every size. A deterministic rule-based baseline reaches 84.6 on Tier 1, with the remaining language-model advantage concentrated in policy adaptation, compound scenarios, accessibility, and temporal reasoning. Muse Glimmer 30B leads the composite ranking, and serving configuration alone moves the Qwen 3.5-to-3.8 comparison by 2.7 Tier 1 points. The benchmark, harness, reproduction guide, and fine-tuned students are released at https://github.com/continker/metrollm-bench.

0.36

Network-based Spatial Context Retrieval for Open-weight LLMs: A Faithfulness Benchmark for Grounded Geographic Reasoning

cs.LG ·2026-09-30 ·Joan Perez
benchmarkcode/data similarity 0.58
Abstract

Large language models (LLMs) encode substantial latent geographic knowledge, yet they reason poorly over space and are unreliable when queried from coordinates alone. Useful behaviour emerges only when structured spatial context is supplied in the prompt. This raises a question geographic evaluation has left unexamined: once the right context is supplied, does the model reason from it, or override it with its own parametric recall? We take up this question with an open pipeline for network-based spatial context retriev-al. In it, the surroundings of a selected point are defined by the pedestrian street network, the area actually reachable on foot. Using only open data and open-weight models, the pipeline retrieves features from OpenStreetMap and the GHS-POP population grid, computes indicators over the network catchment in code, and injects them as a compact spatial brief. On this basis we build a faithfulness benchmark. It labels every claim a model makes by its source (grounded in the brief, or drawn from training knowledge) and its correctness, and it probes each case with a planted false premise that the brief refutes. We evaluate sixteen open-weight model configurations across three families (Qwen, Gemma and Llama, with Gemma in two generations), four size classes and, where available, both thinking and non-thinking modes, on three con-trasting cities, resampling every case over ten seeds. The results show that resistance to the planted premise varies more strongly by model family and generation than by scale, while brief-reading competence forms a partly separate dimension. These behaviours are not captured by conventional world-correctness scores or single-shot evaluation. We release the implementation, spatial briefs, model outputs, and claim-level labels as a reproducible workflow at github.com/perezjoan/NSCR-LLM.

0.27

Fine-tuning LLMs for Tourist Trajectory Prediction using Field Experiment Data

cs.CY cs.LG ·2026-08-21 ·Tatsuya Amano, Hirozumi Yamaguchi
published similarity 0.58
Abstract · 5 pages, 4 figures. Accepted at the 2nd Workshop on AI for Urban Planning (AI4UP) at AAAI-26, Singapore, January 2026

Evaluating mobility interventions at tourist destinations requires predicting visitor behavior under varying conditions. Traditional methods struggle because tourist decisions depend heavily on context like weather and fatigue, yet models cannot generalize to unobserved scenarios. Large Language Models offer a solution by encoding commonsense knowledge about human behavior from pretraining, enabling reasoning about context-dependent decisions, while natural language representation flexibly integrates heterogeneous information. Fine-tuning on local trajectories adapts this general understanding to destination-specific patterns. We validate this approach using 566 trajectories from Wakayama Castle Park, Japan. Our fine-tuned Llama-3.1-8B achieves 49.1% next POI accuracy and maintains strong performance on undersampled scenarios like rainy days, demonstrating effective generalization. This establishes LLMs as high-fidelity behavior models for context-dependent tourist prediction, providing groundwork for counterfactual analysis of mobility interventions.

0.33

Beyond the Desert Label: A Pathway Diagnostic for User-Centered Smart Mobility Service Design

cs.CY ·2026-09-18 ·Oluwasegun Adegoke, Sevgi Erdogan
similarity 0.58
Abstract · 5 pages, 4 figures. Accepted for publication in the Proceedings of the 2026 IEEE International Smart Cities Conference (ISC2), Workshop W02,…

Smart-city mobility platforms increasingly rely on spatial screening tools to identify neighborhoods where public transit fails dependent users, but a single transit desert label can mask very different user problems: localized mismatch between service and concentrated need, or basic absence of usable service. These call for different user-centered responses. This paper introduces a pathway-based, reproducible, data-driven diagnostic that distinguishes relative transit mismatch from minimum-service failure and reports the specific service attributes (frequency, span, weekend service, walking access, and destination accessibility) driving each classification. The workflow combines open data (GTFS, ACS, LEHD, Census, and OpenStreetMap), detects spatially coherent mismatch using Local Moran's I, and applies an equity-informed service-failure test that centers vulnerable users. Applied to Baltimore, Philadelphia, Nashville, and Dallas, the diagnostic shows that legacy-transit cities are dominated by localized mismatch, while auto-oriented cities show broader minimum-service failure, with distinct service-deficit profiles in each case. By making the mechanism behind an under-service label explicit, the tool supports more inclusive, user-centered smart-mobility planning across cities with different transit baselines.

0.25

CALM: A Calibrated LLM Choice Network Framework for Activity-Based Traveler Simulation

cs.LG cs.AI ·2026-09-06 ·Yezhou Cheng
similarity 0.57
Abstract

We present CALM, a reproducible hybrid framework that integrates an optional large language model (LLM) activity planner with calibrated stochastic choice, shared network feedback, memory and habit, typed feasibility checks, and deterministic offline replay. Unlike trip-mode classifiers or diary-only generators, CALM executes a closed traveler-day loop and evaluates each generative module against an empirical, reproducible baseline. On the 2024 New York City Citywide Mobility Survey (CMS), 110,691 seven-mode trips are split by respondent into 78,487 training and 32,204 holdout trips. Training-only alternative-specific constant calibration reduces mean holdout mode Jensen-Shannon divergence from 0.15599 to 0.00394 across ten seeds. A matched live-LLM ablation then quantifies trade-offs among aggregate fit, temporal fit, behavioral persistence, and feasibility, while frozen prompt-response pairs support deterministic replay of downstream simulation. Controlled weather, delay, fare, and parking ladders further demonstrate consistent and interpretable responses under intervention. CALM contributes a reproducible protocol for integrating and evaluating generative planners in traveler simulation through person-disjoint calibration, matched module ablation, controlled stress testing, and end-to-end traceability.

0.14