CS Paper Radar

SignalAudit-QD: A Reproducible Audit Resource for External Signals in AI-Query Forecasting

Amir Rafe, Iman Jamshidi, Gourab Sil, Subasish Das

cs.IR ·submitted 2026-11-12 15:25 UTC · v1

Abstract

SignalAudit-QD is a reproducible audit resource for testing whether a dated public signal adds information beyond query history in a fixed 20-term U.S. AI panel. It aligns two archived Google Trends extractions with eight USPTO AI-patent publication streams and releases origin-indexed predictions, paired losses, provenance, executable tests, and tamper-evident run manifests. Both inputs are retrospective reconstructions obtained after the modeled 2020--2023 origins; the task is therefore a pseudo-origin assay within frozen measurement vintages, not a historical deployment simulation. In the designated one-month comparison, adding a two-component patent block to a level ridge changes macro mean absolute error by +0.0012 Trends points, with conditional 95% moving-block interval [-0.0363,+0.0387]. This establishes neither aggregate benefit nor equivalence. Alternative dimensions, semantic mapping, patent definitions, a second extraction, and a three-month horizon remain unresolved; persistence-anchored and nonlinear hosts instead show model-specific worsening, tiny for the anchored fit. Matched timing controls expose availability assumptions, while a planted-signal assay detects only the largest tested injection. A within-query high-interest alert demonstrates the released interface but not online utility. The contribution is a bounded negative case study and an extensible evidence schema for auditing external-signal claims in temporal information retrieval.

Closest papers

Nearest neighbours by meaning across the whole index.

Learning and Predicting Patent Technology Reuse Trajectories from Emergence-Time Signals

cs.CY cs.ET ·2026-09-30 ·Ayham Yousef, Qiang Ye, Qiang Cheng
benchmark similarity 0.60
Abstract

Forecasting how a newly emerged patent technology will be reused is central to technology intelligence, but reuse-pattern labels do not exist in advance: they must be constructed from the trajectories themselves, and how they are constructed determines what a forecast means. We study $201{,}710$ novel patent technologies (first-time IPC code pairings, USPTO 2002--2022). Our primary labeling applies $k$-means in the latent space of a GRU autoencoder trained on the $20$-year reuse trajectories, using no hand-crafted features; to our knowledge this is the first use of a learned sequence representation for this task. Seven emergence-time features, observable in a technology's first year, recover these labels at a one-vs-rest macro ROC-AUC of $0.914$, but the calendar year of emergence alone reaches $0.874$. A replication on technologies observed for ten full years, none of them right-censored, indicates that this calendar-year effect mainly reflects change over time in what was patented. Separately we cluster the emergence-time features themselves, after Fractal Autoencoder feature selection. That emergence-profile partition agrees with the GRU-based labels only marginally above chance (Adjusted Rand Index $\approx 0.04$), so a partition of emergence-time features is not a reuse-pattern taxonomy and should not be read as one. On a trajectory-shape task following the published construction, GBDT reaches $0.831$ with all seven features and $0.740$ with the selected subset; the published $0.728$, from a different corpus and labeling, is a reference point rather than a benchmark. Two features are additionally left-truncated for the earliest cohorts, which we quantify.

0.34

Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence

cs.AI ·2026-08-14 ·Fabricio F Costa
surveybenchmark similarity 0.56
Abstract · 16 pages, 6 figures, 1 table. Evidence cutoff: 12 August 2026. Ancillary code and data included. Preprint; comments welcome

Quantitative forecasts of frontier artificial intelligence often connect dated targets to trends in benchmark scores, training compute, release time, or expert belief. This paper audits whether the public measurement record supports those connections before another trend is fitted. I construct a frozen, event-centric record through 12 August 2026 with 62 selected systems, 12 versioned benchmarks, seven capability or impact criteria, 144 graded events, 27 source records, and 408 typed relations. The record is an audit sample, not a census. Only seven systems jointly observe estimated training compute and a METR 50 percent task horizon. Training compute is absent for 19 of 27 closed systems, including every selected closed release from 2026, while none of the 35 open-weight systems has a METR horizon observation. Benchmark succession creates a second break: a seven-system link from METR Time Horizon 1.0 to 1.1 has a log-scale slope of 1.206 (95 percent CI 1.021 to 1.390), whereas a six-system MMLU to MMLU-Pro comparison appears shift-like under logit and probit links but not under linear or logarithmic links. The observed bridges have about 80 percent power only for slope departures near 25 percent. Provenance is concentrated: 52 of 71 substantive quantitative events, or 73.2 percent, come from one measurement programme, and 76.1 percent are laboratory releases. A review of 56 methodological and empirical sources identifies 16 complementary measurement directions spanning resources, inference budgets, reliability, agentic work, safety, human preference, field outcomes, and forecast backtesting. No direction supplies a replacement scalar. The result is not that frontier AI forecasting is impossible, but that a defensible dated forecast is a claim about a versioned measurement system with explicit joins, protocols, links, and source dependence, not merely a fitted curve or calendar date.

0.12

CAME: Company-Aware Evidence-Memory Experts for Interpretable Quarter-Ahead Revenue Forecasting

cs.CL cs.LG ·2026-09-27 ·Ya-Wen Wu, Meng-Fen Chiang, Kuang-Da Wang, Wen-Chih Peng
top venuehot direction similarity 0.54
Abstract · Accepted to EMNLP 2026 Main Conference

Quarter-ahead revenue forecasting requires company-scale numerical accuracy, strict temporal validity, and company-specific interpretation of narrative disclosures. LLMs can distill textual evidence but can produce scale-misaligned forecasts, whereas history-based anchors are stable but miss forecast-time signals such as product transitions, supply constraints, and management guidance. We introduce CAME (Company-Aware Evidence-Memory Experts), a residual-forecasting framework that refines a no-leakage statistical anchor when current semantic evidence and prior error patterns justify an adjustment. On a development-inclusive rolling backtest of 336 company-quarters from 12 large public technology and platform firms, CAME achieves the lowest aggregate point-estimate error among the reported methods, with statistically supported macro-sMAPE gains over the matched Statistical Anchor, and outperforms History + Guidance on all six aggregate metrics. CAME also links adjustments to source-linked evidence cards and guarded memory traces, supporting forecast inspection, provenance, and failure localization.

0.55

Do Time-Series Foundation Models Pay Off for Industrial Monitoring? A Cost-Aware Empirical Study

cs.LG ·2026-08-24 ·Guan-Hua Wen, Kuan-Yu Chen
similarity 0.51
Abstract · Accepted for poster presentation at CIF26 (2026), Poster P01013. 7 pages, 2 figures

Industrial monitoring models must detect operationally relevant deviations while satisfying target-specific data, calibration, and resource constraints. Time-series foundation models (TSFMs) promise reusable representations and zero-shot forecasts, yet evidence for their deployment value remains mixed when task definitions are heterogeneous and lightweight baselines are competitive. This work presents a protocol-aware empirical assessment across three settings: a C-MAPSS degradation-risk proxy, normal-only training for anomalous-sound detection on MIMII, and BDG2 forecasting-residual diagnostics with synthetic target perturbations. We assess classical one-class methods, compact neural autoencoders, residual forecasters, MOMENT-small, Chronos-T5, and TimesFM 2.5 in terms of anomaly-ranking performance, risk-horizon sensitivity, residual forecasting and perturbation sensitivity, and local implementation cost. Across 100 C-MAPSS engines evaluated out of fold, TCN-AE reaches fold-weighted AUROC/AUPRC 0.9570/0.8960, compared with 0.7310/0.3080 for MOMENT reconstruction; paired engine-cluster bootstrap confidence intervals exclude zero for both differences. Across five matched MIMII pump evaluations, OCSVM also exceeds MOMENT reconstruction in AUROC and AUPRC. On a fixed 12-meter BDG2 panel, TimesFM 2.5 has the lowest aligned forecast error and the highest synthetic AUROC point estimate, although synthetic AUPRC is similar across TSFM and fitted residual models. Same-device measurements show that MOMENT incurs higher latency, peak allocated VRAM, and serialized state-dictionary size than TCN-AE. Under the evaluated frozen and zero-shot settings, TSFMs are task-dependent deployment options rather than default replacements for fitted lightweight models.

0.34

ArtifactBench: Lineage-Aware Evaluation of AI-Generated Music Detectors under Distribution Shift

cs.SD eess.AS ·2026-09-20 ·Heewon Oh
benchmarkhot direction similarity 0.51
Abstract · 9 pages, 1 figure, 6 tables. Related to ArtifactNet (arXiv:2604.16254); this work contributes the benchmark and evaluation protocol rather t…

AI-generated music detectors are commonly compared using aggregate scores on benchmarks whose training overlap, generator lineage, source provenance, and audio-transformation history are only partially observable. This paper introduces ArtifactBench, a lineage-aware evaluation suite for measuring detector behavior across generator families and versions, real-music domains, collection-cohort shift, and inference coverage. The benchmark groups source recordings and their derived variants by content identity, separates calibration from final testing, records inference failures independently from classification errors, and reports source-level performance with uncertainty in addition to aggregate metrics. We evaluate multiple publicly available detectors under a version-pinned common protocol and examine how leakage control, cohort availability, threshold policy, and model-specific missingness alter measured performance and model ranking. On the 562-track common-success test intersection, ArtifactNet obtains 0.982 AUROC and 0.918 balanced accuracy, compared with 0.761/0.776 for the public Deezer detector; SpecTTTra and CLAM fall below 0.30 AUROC under this shifted cohort. These results also expose substantial generator- and real-domain shifts that aggregate scores alone conceal.

0.39

The AI Risk Observatory: What Can We Learn from AI Disclosures in Annual Reports About Societal Resilience?

cs.AI ·2026-10-01 ·Bart Jaworski, Andrew Sutton
code/data similarity 0.51
Abstract · 22 pages (9 main text + appendices), 12 figures, 7 tables. Code and data: https://github.com/84rt/AI-Risk-Observatory (release dataset-v1.1)

Societal resilience research relies on access to useful and actionable data, which motivates our main research question: Can annual reports, processed at scale with LLMs, provide a useful signal about how companies disclose their response to AI? We test this by applying a reproducible two-stage classification pipeline to 9,821 annual reports from 1,362 UK listed companies (2020-2025, with partial 2026 data). We first validate the method against 474 human-annotated passages, finding high recall and moderate label-level agreement. We then report three empirical patterns: (i) between 2020 and 2025, the share of reports mentioning AI risk rose from 2.8% to 41.2%, while AI adoption disclosure also rose, from 13.8% to 45.2%, and named vendor mentions cluster around a small set of major providers led by Microsoft; (ii) disclosure varies substantially by Critical National Infrastructure sector and market segment: AIM reports disclose AI risk at far lower rates than Main Market reports, and sectors such as Energy and Data Infrastructure lag behind the rest in AI risk disclosure; and (iii) harm disclosures are near-absent (seven reports across the entire corpus). We develop a substantiveness classification to assess the quality of the disclosure and find that most AI risk disclosure is not substantive: in 2025, 41.2% of all reports mention AI as a risk, but only 4.3% contain AI risk disclosure we classify as substantive.

0.32

TimeSage-EV: A Live Benchmark for Agentic Time Series Analysis in Evolving Environments

cs.AI ·2026-08-14 ·Qingren Yao, Yaxuan Kong, Yuqi Nie, Yichen Li, Stefan Zohren +4 more
top venuebenchmark similarity 0.50
Abstract

Time series analysis in high-stakes domains relies on recurring data releases, where new observations can alter the evidence base and the validity of later conclusions. Existing time series QA benchmarks mostly rely on fixed snapshots, leaving temporal validity and cutoff-aware evidence use unevaluated. We introduce TimeSage-EV, a live benchmark for agentic time series analysis in evolving environments. It tracks 60 real institutional scenarios across 6 domains, comprising 1,485 scenario-period QA pairs from Feb 2023 to May 2026 and spanning monthly, weekly, daily, and irregular release cadences. At each period, large language model (LLM) agents receive time series data and source reports, while the withheld target release provides ground truth. TimeSage-EV evaluates state identification, data summarization, and outlook reasoning. Experiments with frontier LLM agents and TimeSage-1.0, a novel self-evolving agent with a reusable analytical skill library, reveal significant performance gaps across model tiers and recurring failures in temporal validity, exogenous context use, and adaptation. We release TimeSage-EV as a research resource with monthly updates, code, a leaderboard, and failure-mode analyses.

0.34

Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation

cs.AI cs.IR ·2026-09-10 ·Koutian Wu, Wenjun Zhang, Junjie Zhou, Pengqian Han, Qiming Bao +11 more
benchmarkcode/data similarity 0.50
Abstract · Code: https://github.com/ktwu01/benchmark-radar, Project site: https://benchmark-radar.org

Benchmark researchers and developers of large language models (LLMs) and other AI systems need to find relevant evaluations, locate their benchmark datasets and code, and understand the settings behind reported scores. We present Benchmark Radar, a living database and search engine for retrieval and discovery of AI benchmarks, covering LLM evaluation, agentic and tool-use benchmarks, coding, reasoning, safety, and domain-specific evaluations. The system combines daily discovery of benchmark papers, repositories, datasets, and releases with a searchable benchmark catalog, mentions in model cards and technical reports, and score histories. It retains source identities and citations so readers can inspect candidate benchmarks and their evaluation evidence. Daily discovery draws on 37 sources: 13 direct connectors and 24 first-party research and engineering feeds. The catalog contains 1,283 source records drawn from 4 benchmark catalogs and 12,916 numeric observations on 790 records. We describe collection and retrieval, audit the full catalog, and examine benchmark saturation, adoption trends, and the limits of score comparisons. A worked example walks through a complete prior-art search, showing how to query the catalog and inspect benchmark evidence when designing a new evaluation. We release the web dashboard with a benchmark leaderboard, a Pareto frontier view of score against measured use, saturation and trend views, daily feeds, downloadable evidence, a command-line interface (CLI) for offline queries, and reproducible analysis.

0.37