CS Paper Radar

Who sets the consent signal? Platform defaults and crawl coverage in measuring opt-outs from AI training

Marcin Lawenda

cs.NI ·submitted 2026-11-13 14:18 UTC · v1

Abstract

Web data used in social research and in the training of AI models increasingly carry consent signals, but who sets these signals is rarely measured. This paper attributes opt-outs from AI training to the party that set them. The complete lists of Polish, Czech, Slovak, Hungarian and German domains recorded by Common Crawl, with 3,445,751 reachable sites, were crawled in October 2026 together with a random sample of 10,000 Slovak registry domains. Each signal was matched against vendor templates and identical files. Four exploratory hypotheses were examined with logistic models. Between 0.70% and 3.82% of the listed sites opt out. Platform and hosting defaults set 15% to 78% of the opt-outs. Settings active by default reach 60% to 95% of a vendor's sites, while settings that owners switch on reach at most 2.1%. Most of the higher Czech and Slovak rates compared with Germany are statistically attributable to platforms. Lists derived from Common Crawl miss about a third of the Slovak opt-outs because they exclude sites that block its crawler. The results describe delegated consent, a signal that a platform sets on behalf of its users, which measurements of consent on the web must consider.

Comment
24 pages, 6 figures, 2 tables. Dataset: https://doi.org/10.5281/zenodo.23686009

Closest papers

Nearest neighbours by meaning across the whole index.

Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls

cs.CR ·2026-11-04 ·A H M Nazmus Sakib, Mahsin Bin Akram, Raveen Wijewickrama, Igor Bilogrevic, Murtuza Jadliwala
similarity 0.63
Abstract

Web tracking studies largely rely on automated crawlers, yet their measurements can differ from what people encounter while browsing. Conventional crawlers follow predefined navigation rules, whereas LLM-driven agents decide how to navigate and interact with websites, potentially changing the tracking they encounter. We compare agentic and non-agentic crawlers to examine how websites respond to them, what tracking they record, and how closely their observations match human browsing. Across 2,000 popular websites, we evaluate different navigation strategies, automation frameworks, and stealth configurations, alongside browsing data from 26 participants. We find that website denials increase beyond landing pages for standard browser automation, while a detection-resistant automation framework largely prevents this escalation without increasing recorded tracking. On the same URLs, agentic crawlers record tracking activity from 1.5-1.7 times as many third-party organizations, particularly on consent-managed pages. Agentic crawlers recover 82-87% of tracker organizations observed by participants, compared with 58-68% for non-agentic crawlers, yet roughly half of their own observations are absent from human participant visits. These findings show that avoiding website interventions, increasing tracking visibility, and achieving closer agreement with human observations do not necessarily go hand in hand, highlighting the need to select and validate crawlers according to the measurement objective.

0.17

ScrapeBench: Evaluating Legal Compliance of AI Agents in Website Scraping

cs.CL cs.AI ·2026-11-08 ·Joseph Marvin Imperial, Daniel Slate, Noam Kolt
top venuebenchmark similarity 0.63
Abstract · Camera-Ready for NeurIPS 2026 (Evaluations and Datasets Track)

AI agents deployed on the open web are becoming more capable of performing actions that may breach contractual provisions and violate computer security laws. In this paper, we introduce ScrapeBench, the first agentic benchmark for empirically evaluating the legal compliance of AI agents with website scraping policies, namely terms of service (ToS) and robots$.$txt directives. Across widely used closed-weight and open-weight AI agents, we find that most agents exhibit low compliance rates (below 5%) when instructed to scrape websites in our sample of 1,626 popular websites that explicitly prohibit scraping. In addition, we observe that even where AI agents actively check the applicable ToS and robots$.$txt, they continue to exhibit low compliance rates. Conversely, in our sample of 269 websites that do not prohibit scraping, we observe that agents sometimes refuse to scrape despite it being lawful, such as Claude Opus 4.6 exhibiting an over-refusal rate of 98.5%. Our human baseline experiments (n = 180 participants) suggest that humans exhibit compliance rates comparable to AI agents but over-refuse at much higher rates.

0.51

One String to Rule Them All: Measuring Consent Under the IAB Global Privacy Platform

cs.CR ·2026-11-01 ·Muhammad Abu Bakar Aziz, Christo Wilson
similarity 0.63
Abstract · 11 pages, 8 figures, 3 tables

Many privacy regulations, including the California Consumer Privacy Act (CCPA), require websites to honor users' opt-out choices. To facilitate this process, industry has introduced technical standards such as the IAB Global Privacy Platform (GPP), which aims to standardize collection and transmission of user consent information across multiple jurisdictions. However, it remains unclear whether these standards consistently and correctly transmit user opt-out choices, or whether opt-out choices are respected by advertisers. In this study, we conduct an in-depth analysis of consent propagation under the GPP framework. We collect data from four web crawls across four experimental settings to analyze how widely the standard is adopted, the extent to which opt-out signals propagate from publishers to third parties, and whether these signals reach and alter third-party advertiser behavior. Our analysis reveals several inconsistencies in the implementation and adoption of the standard that result in user consent signals being lost and user opt-out choices not being respected by the advertisers.

0.28

A Critical Appraisal and Motivated Intruder Test of A Search Dataset Anonymised for the Purposes of the EU Digital Markets Act

cs.CR ·2026-11-16 ·Ghislain d'Entremont, Lisa Pilgram, Sophie Stalla-Bourdillon, Renee Gil, Lucy Mosquera +2 more
similarity 0.59
Abstract

This study evaluated the European Commission's anonymisation measures for Alphabet's search-data sharing under Article 6(11) of the Digital Markets Act against European Data Protection Board guidance and contemporary practice, and assessed re-identification through motivated intruder testing. The measures focus on search end-users, leaving other individuals mentioned in queries insufficiently protected, and do not explicitly address inference. The accompanying controls and safeguards reflect good practice for managing residual risk. Motivated intruder testing targeted 98,501 individuals across six countries using publicly available profiles. Google-verified matches yielded a re-identification rate of 0.094%, below the study's 0.47% per-team threshold. Each verified match required about three hours (Euro 105), excluding Google verification and Gemini costs, suggesting limited cost deterrence. All country-level rates were below the threshold, but one region exceeded it. The threshold drew on conservative precedents, adjusted for data quality, verification, controls and safeguards, assuming five independent attack teams. Some sensitivity scenarios exceeded the threshold, but relied on strong assumptions, such as treating all uncertain matches as correct. Results may not generalise beyond the targeted population, and ethics requirements constrained the attacks. Personal data detection recall was assumed rather than tested: the minimum defensible recall was 0.88, with 0.976 recommended. However, missed identifiers outside the allow-list would remain in the dataset. Under the study's contextual framework, assumptions and testing limitations, identity-disclosure risk was considered insignificant. This finding does not resolve the gaps concerning inference and other individuals' information; safeguards remain necessary.

0.32

Drowning in AI Slop: How Social Media Platforms (Do Not) Label AI and Deepfake Content under EU law

cs.CY ·2026-09-29 ·Bram Rijsbosch, Luka Bekavac, Henry Tari, Gijs van Dijck, Konrad Kollnig
similarity 0.54
Abstract · This work is under submission and has not undergone peer review yet

AI labels are emerging as a primary safeguard for transparency about AI-generated content on social media, including under the EU Digital Services Act and AI Act. Yet, limited systematic evidence exists on how platforms implement such labels in practice. We conduct a legally grounded audit of AI labelling across Instagram, TikTok, X, and YouTube, drawing on the European Commission's July 2026 guidelines on deepfakes. To this end, we analyse platform policies and detection approaches, 10,722 posts collected via systemic-risk and AI-related keywords, an expert-annotated subset of 500 posts, and controlled uploads to the four platforms of outputs from ten popular generative AI tools. We find that AI labelling is now broadly established. All four platforms apply labels automatically and a greater share of labels are platform-applied than in earlier audits. However, coverage remains incomplete where labelling arguably matters most: only 33% of expert-identified deepfakes in systemic risk contexts carried a platform-applied AI label, while reaching a median of 160,000 views. In controlled uploads of AI-generated content carrying standard AI provenance signals, platforms labelled only 61% of uploads, and commonly strip those signals after uploading. Overall, platform rules, label designs, detection approaches, and reporting diverge substantially. In response, we identify concrete opportunities to improve the uptake, clarity and efficacy of AI labelling.

0.05

User First Intelligence: Three Measurements of Search and AI-Answer Analytics (SEO and GEO/AEO), and a Knowledge-Graph Framework for Query Discovery Beyond Search Volume

cs.IR ·2026-11-14 ·Venkata Pagadala
code/data similarity 0.54
Abstract · 43 pages, 10 figures. Preprint, not peer reviewed. Data and scripts in the ancillary files; implementation at https://github.com/VenkataPaga…

People say what they want to know in many places besides a search box, and the tools that search and AI-answer optimization (SEO and GEO/AEO) rely on see only part of that demand, through models reported as counts. This paper reports three measurements of such instruments and a framework. On the area-code section of a large US telecom provider's site, two commercial analytics platforms differed on estimated monthly visits by 7.5x to 27.1x (8.6x to 29.9x worldwide); across twelve sections of the domain the ratio ran from 0.20x to 7.24x with no detectable relation to section age, while they agreed on which section is larger in 60 of 66 pairs. On 37,193 reviews of 378 telecom stores linked to a 42,994-node keyword taxonomy, review demand and search volume agreed on branches to a varying degree (Spearman 0.47 to 0.75; 0.91 post hoc) and barely on topics (0.18), on links two blind AI annotators judged two-thirds correct; on the 372 area codes, a public count of Wikipedia requests for each code and vendor volume agreed roughly on ranking (0.45 over 240 articles, gone once keywords matched per code are held constant) and not on magnitudes, predictions fixed first. In one vendor's index of AI answers naming or citing the largest companies of each S&P 500 sector, of four predictions written first, one held, one in direction only and two failed; video and community sites took a median 21% of citation incidences on Google's AI Overviews and AI Mode against 0.6% to 9.2% on four other surfaces, in every sector, and in education answers in each of 13 countries read later. Estimates and sources need measuring per instrument and surface. User First Intelligence (UFI), built on that premise, collects what people ask beyond the website, links it to an industry ontology and scores where an organization does not answer; an open implementation is released, its score not evaluated. No causal claim is made.

0.36

You Get What You Sample: Evaluating Sampling Strategies for Web Security Measurements

cs.CR ·2026-09-10 ·Xuenan Zhang, Yuqing Yang, Giancarlo Pellegrino
published similarity 0.53
Abstract

Web measurement studies rely on domain datasets such as Tranco to quantify the prevalence and impact of security issues at scale, but exhaustively analyzing these datasets is often infeasible because of the cost of advanced analysis techniques, requiring the use of sampling. Despite its widespread use, sampling remains largely guided by convention---most commonly \emph{Top $N$} domain selection---rather than evidence, and its influence on the validity and generalizability of security findings has received little systematic evaluation. Consequently, it remains unclear whether common sampling strategies introduce systematic bias, distort observed vulnerability rates, or limit comparability across studies. In this work, we undertake, to the best of our knowledge, the first comprehensive investigation into how sampling methodologies affect the measurements and the conclusions. Through a comprehensive literature review and large-scale measurements of 500k Tranco and 24.8M Common Crawl hosts, we perform a comparative evaluation of datasets and sampling strategies. We show that, while Top $N$ sampling may be a rational strategy, the researchers have to bear in mind that Top $N$ does not reflect the overall distribution of the web. Instead, probability-based strategies yield stable, unbiased estimates for prevalence and many impact objectives. Hybrid sampling provides no advantages over pure probability sampling, as its deterministic prefix consistently contributes negatively to accuracy. Building on these results, we provide data-backed guidance for future studies, proposing to use an adaptive probability-based sampling strategy that remains effective even when the prevalence of the target issue is unknown.

0.34

The uneven rise of AI-generated web content

cs.AI cs.CYcs.NIstat.AP ·2026-10-27 ·Adam Visokay, Soumyakanti Pan, Molly Griston, Polina Iasakova, Mark Nepf +8 more
similarity 0.52
Abstract · 12 pages, 4 figures

In less than five years, generative artificial intelligence (AI) has dramatically reshaped the public-facing web. Using a sample of more than 5 million crawlable webpages, we demonstrate that the share containing AI-generated text has increased precipitously, but unevenly. Our analyses reveal substantial differences in AI composition across language and domain categories (e.g., .edu, .com, or .gov). A higher fraction of English-language webpages contain AI-generated text compared to pages in other languages, and prevalence is substantially lower on .gov and .edu domains compared to .com. These distinctions persist even after accounting for compositional effects. We also find heterogeneity based on a page's position in the connectivity structure of the web. AI-generated text is most common and has increased most rapidly in pages that are the least central in the graph, which drives much of the overall growth in AI-generated text. A slower increase has taken place for more central pages, with the most central tier emerging as the least likely to contain AI-generated text. Methodologically, we use multiple AI detection tools, and generate estimates using a Bayesian hierarchical model that incorporates corrections for the detectors' statistical properties (both false positives and false negatives) and allows for different detector performance across groups, while propagating uncertainty in detector error rates into prevalence estimates. Our work suggests that AI is a prominent, but varied, contributor to public information and internet-based communication.

0.17