CS Paper Radar

Do Open-Source Video Ad Clients Follow VAST 4.3? An Empirical Study of Player Conformance

Prashant Chaudhary

cs.SE cs.MMcs.NI ·submitted 2026-10-29 00:34 UTC · v1
code/dataunusual

Abstract

Context. Video ads reach streaming and connected-TV apps through VAST documents, and companies count ads from the HTTP requests the player's client library sends. A tracking error can leave playback intact and surface later as disagreeing counts. Objective. To measure how far open-source VAST clients meet the player duties of VAST 4.3, whether verdicts depend on the runtime, what generated inputs add, and when deviations were introduced. Method. A catalog of 108 requirements was extracted from the specification. Six open-source clients in eight configurations ran against 69 crafted test vectors and the IAB's sample tags, under Node.js and in five browsers on three engines, with every request recorded. Grids, fuzzing and a sweep over 36 releases extend the catalog; Google's closed IMA SDK is a comparison. Results. The open-source clients have 78 findings and 3 to 21 must-level deviations per configuration. None of the tested open-source clients that follow redirect (Wrapper) chains fills the required macros in a Wrapper's request; the archived IMA 3.794.0 configuration tested does. One library's timeout throws an exception no caller can catch, ending a Node.js process. In a pod (ads played back to back), one player credits a finishing ad's completion to the next ad, fixed in release 5.3.0. A library can conform under jsdom and deviate in every tested Chrome- and Safari-family browser. Conclusions. The tested clients diverge on the duties other companies use to count and diagnose ads. A shared conformance suite run in real browsers could expose most graded deviations; the harness and vectors are released.

Comment
51 pages, 20 tables, 3 figures. Harness, test vectors and results: https://doi.org/10.5281/zenodo.23581408

Closest papers

Nearest neighbours by meaning across the whole index.

Contain, Conform, Compare (Extended Version): A Red-Teamed Evaluation Kit for Small Coding Agents and the Full Record of Its Experiments

cs.AI ·2026-11-12 ·Nguyen Minh Thang
code/data similarity 0.51
Abstract · 52 pages, 15 figures, 37 tables. Code: https://github.com/MinhThanghus/drydock-eval (tag v0.4.0). Data and analysis scripts are included as …

A hosted score in the Gemma 4 Developer Agent competition is one noisy number, so development needs a local evaluator that is fast, safe for agent-written code, and matching the hosted scorer's prompts. drydock-eval evaluates a small coding agent (Gemma 4 E4B) at about 250 runs per hour on one RTX 5090. Contain runs each job in a fresh container without network and passed 200 scripted escape tests; conform provides eight checks that compare a local stack's requests with the scorer's configuration, down to rendered tokens; compare provides paired statistics that respect repository clusters. AI agents built and attacked the kit; a human owner approved machine use and release. This extended version repeats the short paper and adds the full record of the experiments made with and around the kit: 9,660 rows of agent runs, 2,494 grading controls and 72 serving measurements, all released. On four L4 GPUs the 31B model decodes 38.1 tokens per second in one stream and E4B 105.8. Six configurations of a 31B agent solved between 37 and 44 of 70 development tasks, and no pairwise difference was detected. A pre-registered five-arm experiment (231 tasks, 4,620 jobs) measured four prompt divergences: no E4B solve-rate change was detected (largest estimate -2.1 points, 95% interval -4.4 to +0.4). Three changes of the E4B agent led their screens by +1.0 to +6.2 points; each was then tested once on new tasks under a frozen pass rule, and none was confirmed (gains -0.35, -0.39 and +4.10 points). The unchanged baseline, run again on the same 128 tasks, moved by 1.17 points, and 65% of these tasks were solved in none of eight runs. Prompts and agent bundles are withheld; every run-level result is public.

0.53

EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications

cs.AI cs.SE ·2026-10-12 ·David Nukrai, Bar Alon, Nativ Levy, Ido Andrew Atad, Ido Cohen +4 more
benchmarkcode/data similarity 0.51
Abstract · 24 pages, 6 figures. Code: https://github.com/two-delta/EvoWeb ; Dataset: https://huggingface.co/datasets/TD-Labs/EvoWeb

Web application development often proceeds through successive requests that extend an existing implementation. Yet web-generation benchmarks primarily evaluate development toward fixed specifications. We introduce EvoWeb, a novel benchmark comprising 320 cumulative requests over 32 web applications, grounded in real-world product features and structured by functional dependency graphs. Agents iteratively modify a persistent codebase, while a multimodal web agent tests each intermediate application in a live browser, assessing newly requested functionality, cross-request integration, and previously requested functionality. To characterize reliability throughout development, we integrate trajectory performance across user-tolerance thresholds rather than impose a fixed stopping criterion. Experiments across open- and closed-source frontier models show that EvoWeb reveals clear performance differences between models, with cross-request integration emerging as a key challenge. Further analysis shows that later requests with deeper or more numerous dependencies are solved less accurately, while runtime errors become more frequent over successive requests. EvoWeb provides a testbed for evaluating and improving coding agents in long-horizon, multi-request development.

0.60

SilentProbe: Measuring Silent Failure in Production APIs Used as Agent Tools

cs.IR cs.SE ·2026-08-29 ·Zongrong Li, Shengkun Ye, Feiyou Guo, Zuoyou Dang
code/data similarity 0.49
Abstract · 12 pages, 9 figures. Code and data: https://github.com/Jasper0122/silentprobe

An LLM agent calling a production API cannot distinguish a query that matched nothing from a query the server did not understand. Both return HTTP 200 with a parsable body, no exception to catch and no field to branch on. We ask what predicts which one occurred, and what it does to the agent. Auditing 721,320 parameters across 2,501 independently published OpenAPI documents, we find that 7.5% declare an enumeration and 15.2% declare any machine-checkable constraint at all, while 40.1% of documents state at least one constraint in prose that their schema does not encode. Executing 219 schema-derived perturbations against live commercial endpoints from 27 vendors, reached through a single aggregation layer (Monid) that publishes a schema and returns a run identifier for every call, we find that constraint form, not vendor identity, predicts honesty: machine-checkable constraints yielded an honest error in 111 of 111 cases, prose-only constraints failed silently in 44 of 61 (p = 2e-13). Twelve models across eight families then met these endpoints on ordinary tasks. A vocabulary that the description merely exemplifies was missed by every model on 88 of 88 attempts, while vocabularies written out in full were used correctly 88 to 91% of the time. Running the full agent loop, models detected the resulting silent failure in 12% of cases, repaired it in 0%, asserted a false negative to the user in 41%, and invented a figure in 12%. Promoting the vocabulary into the schema removes the failure, from 88 of 88 to 0 of 89. The fix is one line of schema rather than a better model. Code, schemas, perturbation sets, agent transcripts and per-call run identifiers are released at https://github.com/Jasper0122/silentprobe.

0.60

Open-Source 5G RAN Platforms: A Dual Perspective on Performance and Capabilities

cs.NI ·2026-08-26 ·Maria Katarine Santana Barbosa, Iasmin Gomes, Vinícius Melo, Kelvin Lopes Dias
code/data similarity 0.49
Abstract · This paper has been published in IEEE WCPNS 2025

Open-source implementations of the fifth generation (5G) Radio Access Network (RAN), such as OpenAirInterface (OAI) and srsRAN, have become increasingly relevant for industry and academic research, rapid prototyping, and low-cost private 5G deployments. The state of the art presents different evaluation scenarios for both platforms; however, these are often limited to individual testbed evaluations, end-to-end studies, or partial comparisons between them. In particular, there is a lack of evaluations considering different Software-Defined Radios (SDRs), RAN configurations, user applications, and numbers of real UEs. To address this gap, this paper presents a comparative evaluation of OAI and srsRAN. The study includes a qualitative assessment of supported features and deployment flexibility, followed by a quantitative performance analysis covering radio resource control setup procedures, compliance with theoretical data rates, and real application workloads with diverse quality of service requirements. Specifically, we consider Video on Demand (VoD) for data rate evaluation, Live Streaming (LS) and Cloud Gaming (CG) as latency-sensitive applications. These workloads provide a comprehensive view of each platform capability.

0.63

Characterizing AI Service Traffic for 6G: An Open Testbed and Initial Results from 3GPP SA4

cs.NI ·2026-11-04 ·Imed Bouazizi
similarity 0.49
Abstract · 19 pages

The design of sixth generation (6G) mobile systems requires measurements of artificial intelligence (AI) service traffic under controlled network conditions. The 3rd Generation Partnership Project (3GPP) Service and System Aspects Working Group 4 (SA4) documents such measurements in draft Technical Report 26.870. In this paper, we describe the open testbed used for the study and analyze the results recorded in version 0.6.2 and its companion workbook. The testbed combines network emulation, packet capture, application measurements, and scenario execution against hosted and self-hosted models. The reported uplink to downlink volume ratios span more than six orders of magnitude. A personal assistant agent has a ratio of 0.007 at its local control interface and 0.50 on its wide-area egress. Deterministic agent signaling follows the configured round-trip delay. In a tokenized video experiment with 125 questions per profile, the number of correct answers changes by at most three through 10% independent uplink loss. Larger losses reduce accuracy, but the available repetitions do not establish a tolerance threshold. We also analyze emulator behavior that affects the interpretation of the study. Per-packet jitter introduces reordering, correlated random-loss settings suppress rare loss events, and a Gilbert-Elliott parameter mapping changes the stationary loss rate. The results support characterization by scenario, measurement layer, and versioned network configuration.

0.24

Bridging the Gap: A Longitudinal Analysis of Extended Identifiers in the Post-Cookie Era

cs.NI ·2026-09-02 ·Michael Smith, Riley Grossman, Krzysztof Franaszek, Antonio Torres-Agüero, Pritam Sen +2 more
similarity 0.49
Abstract · 10 pages

As third-party cookies fade because of browser restrictions, the online advertising ecosystem is turning to extended identifiers (EIDs) as an alternative. EIDs are persistent user identifiers, such as hashed email addresses, that are employed to link users across domains and devices. This paper presents a 41-month longitudinal study examining EID usage in over 145 million HTTP header bidding requests sent to six major supply-side platforms (SSPs) from 616,539 websites. Our findings show that EIDs are widely used and are becoming increasingly prevalent in the digital advertising ecosystem, reaching 83.76% of studied websites by May 2025. Our analysis of the 18 popular EID providers that account for 99.42% of all transmitted EIDs in our dataset raises concerns about the readiness of EIDs as an alternative to third-party cookie tracking. In terms of accuracy, only one identity provider consistently recognizes and identifies that the visitor is a self-identified bot crawler, and many providers regularly transmit multiple EIDs for the same visitor. We also identify privacy concerns with EIDs, as 12 of the providers create persistent EIDs that can identify the same user across visits, websites, devices, and months. Finally, we found that 16 providers transmit EIDs on EU websites without user consent.

0.28

ViBench 2.0: A Benchmark for Long-Horizon Software Building

cs.AI cs.SE ·2026-11-03 ·Daniel Furman, Preeya Kirani, Ibrahim Sheikh, Zhen Li, Peter Zhong +1 more
benchmark similarity 0.48
Abstract

On Replit, our agentic engineering platform, users now build entire products without writing code, through successive feature requests to an agent. Yet existing coding benchmarks evaluate single, fully specified tasks, and many are near saturation. We introduce ViBench 2.0, a benchmark for long-horizon software building that we use to select our platform's models. Its 17 full-stack web products, modeled on widely used software, grow over 241 feature requests and are judged only when finished, as a user would judge them: a browser agent runs 544 test plans, and a build passes only if every plan does, under a grading protocol validated to keep this strict score reliable. Evaluating nine models, we find a reliability gap: the three strongest pass up to 98% of test plans yet deliver a fully working product in at most 65% of builds, and no open-weight model delivers one. The gap is closing fast: across three releases of one model family, pass@1 on the public products rises from 3% to 53% while the cost per build falls by more than half. We release eight products with the harness and grader, hold out nine, and report lessons from production use.

0.12

How Developers Configure LLM Inference: A Large-Scale Study of Real-World Practices

cs.SE ·2026-10-20 ·Nada Zine, Matias Martinez, Romain Rouvoy, Clément Quinton
code/data similarity 0.48
Abstract

LLMs are increasingly integrated into software systems, making deployment and inference configuration a recurring software engineering task. Inference backends expose many options that affect model loading, generation, serving, and request processing. Yet little is known about which of these options developers explicitly configure in practice. In this paper, we present a large-scale empirical study of LLM inference configuration practices in open-source software using Hugging Face Transformers, vLLM, and the OpenAI API. We analyze 5,398 GitHub repositories using at least one of these backends and identify 345,117 explicitly configured option occurrences. Our results show that developers use only a small part of the available configuration space regularly. Although 98.8% of the documented backend-specific options appear at least once, 78.9% are configured in at most 10% of repositories using the corresponding backend. Configurations are also small, with a median of 2 options for Transformers and 4 for vLLM and OpenAI, but the combinations of options vary widely: 56.8-61.6% of multi-option signatures appear in only one repository. We also observe recurring option values as well as substantial variation in how values are chosen. Finally, configuration practices differ across application contexts, with option-adoption differences reaching 30.7 percentage points. These findings characterize the configuration surface developers make explicit rather than the importance or optimality of individual options. They provide empirical evidence for prioritizing future studies of inference configuration and for improving backend documentation and configuration support.

0.29