CS Paper Radar

Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls

A H M Nazmus Sakib, Mahsin Bin Akram, Raveen Wijewickrama, Igor Bilogrevic, Murtuza Jadliwala

cs.CR ·submitted 2026-11-04 09:12 UTC · v1

Abstract

Web tracking studies largely rely on automated crawlers, yet their measurements can differ from what people encounter while browsing. Conventional crawlers follow predefined navigation rules, whereas LLM-driven agents decide how to navigate and interact with websites, potentially changing the tracking they encounter. We compare agentic and non-agentic crawlers to examine how websites respond to them, what tracking they record, and how closely their observations match human browsing. Across 2,000 popular websites, we evaluate different navigation strategies, automation frameworks, and stealth configurations, alongside browsing data from 26 participants. We find that website denials increase beyond landing pages for standard browser automation, while a detection-resistant automation framework largely prevents this escalation without increasing recorded tracking. On the same URLs, agentic crawlers record tracking activity from 1.5-1.7 times as many third-party organizations, particularly on consent-managed pages. Agentic crawlers recover 82-87% of tracker organizations observed by participants, compared with 58-68% for non-agentic crawlers, yet roughly half of their own observations are absent from human participant visits. These findings show that avoiding website interventions, increasing tracking visibility, and achieving closer agreement with human observations do not necessarily go hand in hand, highlighting the need to select and validate crawlers according to the measurement objective.

Closest papers

Nearest neighbours by meaning across the whole index.

AgentShield: Fortifying Web Applications Against AI Agents and Agentic Browsers

cs.CR ·2026-10-30 ·Moiz Ali, Nafis Karim, Rownak Islam, Jyotirmay Chauhan, Dhruv Kuchhal +1 more
top venue similarity 0.69
Abstract · To appear in Proceedings of IEEE Symposium on Security and Privacy Oakland (IEEE S&P 2027)

LLM-powered browser agents are transforming web automation by autonomously interacting with websites and executing complex workflows. However, the effectiveness of existing anti-bot defenses against these systems remains unclear. We present a systematic empirical evaluation of modern agentic browsers against widely deployed anti-bot mechanisms, characterizing their capabilities, limitations, and behavioral differences from humans. Our results show that many defenses designed for traditional automation struggle to reliably detect autonomous AI-driven agents. Accordingly, we introduce AgentShield, a behavioral fingerprinting framework that targets the Perception-Reasoning-Action (PRA) loop underlying agentic user clients. By leveraging dynamic web forms that progressively reveal content and capture fine-grained interaction telemetry, AgentShield exposes behavioral artifacts that enable differentiation between human users and autonomous agents. We further demonstrate Extinction-induced Policy Drift (EPD), a novel attack in which repeated obstruction of task completion causes agents to relax their safeguards and engage in unsafe behaviors. Finally, we present defensive prompt injection techniques that exploit agent-specific weaknesses to mitigate emerging forms of autonomous web abuse that rely on content extraction. Our results highlight both the shortcomings of current web defenses against agentic user clients and the feasibility of practical detection mechanisms tailored to this new generation of web automation.

0.37

ScrapeBench: Evaluating Legal Compliance of AI Agents in Website Scraping

cs.CL cs.AI ·2026-11-08 ·Joseph Marvin Imperial, Daniel Slate, Noam Kolt
top venuebenchmark similarity 0.67
Abstract · Camera-Ready for NeurIPS 2026 (Evaluations and Datasets Track)

AI agents deployed on the open web are becoming more capable of performing actions that may breach contractual provisions and violate computer security laws. In this paper, we introduce ScrapeBench, the first agentic benchmark for empirically evaluating the legal compliance of AI agents with website scraping policies, namely terms of service (ToS) and robots$.$txt directives. Across widely used closed-weight and open-weight AI agents, we find that most agents exhibit low compliance rates (below 5%) when instructed to scrape websites in our sample of 1,626 popular websites that explicitly prohibit scraping. In addition, we observe that even where AI agents actively check the applicable ToS and robots$.$txt, they continue to exhibit low compliance rates. Conversely, in our sample of 269 websites that do not prohibit scraping, we observe that agents sometimes refuse to scrape despite it being lawful, such as Claude Opus 4.6 exhibiting an over-refusal rate of 98.5%. Our human baseline experiments (n = 180 participants) suggest that humans exhibit compliance rates comparable to AI agents but over-refuse at much higher rates.

0.51

Bridging the Gap: A Longitudinal Analysis of Extended Identifiers in the Post-Cookie Era

cs.NI ·2026-09-02 ·Michael Smith, Riley Grossman, Krzysztof Franaszek, Antonio Torres-Agüero, Pritam Sen +2 more
similarity 0.63
Abstract · 10 pages

As third-party cookies fade because of browser restrictions, the online advertising ecosystem is turning to extended identifiers (EIDs) as an alternative. EIDs are persistent user identifiers, such as hashed email addresses, that are employed to link users across domains and devices. This paper presents a 41-month longitudinal study examining EID usage in over 145 million HTTP header bidding requests sent to six major supply-side platforms (SSPs) from 616,539 websites. Our findings show that EIDs are widely used and are becoming increasingly prevalent in the digital advertising ecosystem, reaching 83.76% of studied websites by May 2025. Our analysis of the 18 popular EID providers that account for 99.42% of all transmitted EIDs in our dataset raises concerns about the readiness of EIDs as an alternative to third-party cookie tracking. In terms of accuracy, only one identity provider consistently recognizes and identifies that the visitor is a self-identified bot crawler, and many providers regularly transmit multiple EIDs for the same visitor. We also identify privacy concerns with EIDs, as 12 of the providers create persistent EIDs that can identify the same user across visits, websites, devices, and months. Finally, we found that 16 providers transmit EIDs on EU websites without user consent.

0.28

Who sets the consent signal? Platform defaults and crawl coverage in measuring opt-outs from AI training

cs.NI ·2026-11-13 ·Marcin Lawenda
similarity 0.63
Abstract · 24 pages, 6 figures, 2 tables. Dataset: https://doi.org/10.5281/zenodo.23686009

Web data used in social research and in the training of AI models increasingly carry consent signals, but who sets these signals is rarely measured. This paper attributes opt-outs from AI training to the party that set them. The complete lists of Polish, Czech, Slovak, Hungarian and German domains recorded by Common Crawl, with 3,445,751 reachable sites, were crawled in October 2026 together with a random sample of 10,000 Slovak registry domains. Each signal was matched against vendor templates and identical files. Four exploratory hypotheses were examined with logistic models. Between 0.70% and 3.82% of the listed sites opt out. Platform and hosting defaults set 15% to 78% of the opt-outs. Settings active by default reach 60% to 95% of a vendor's sites, while settings that owners switch on reach at most 2.1%. Most of the higher Czech and Slovak rates compared with Germany are statistically attributable to platforms. Lists derived from Common Crawl miss about a third of the Slovak opt-outs because they exclude sites that block its crawler. The results describe delegated consent, a signal that a platform sets on behalf of its users, which measurements of consent on the web must consider.

0.34

Proof-of-Agentness: Towards Verifying Agents on the Web

cs.AI ·2026-10-26 ·Zuyao Xu, Xiang Li
similarity 0.63
Abstract · 12 pages, 7 figures

CAPTCHAs and bot detection have helped protect Web services from automated abuse for decades by distinguishing humans from bots. However, the emergence of LLM-powered agents introduces a new setting: Web services increasingly accommodate agents as legitimate users. Without a way to distinguish agents from other automated clients such as simple bots and scripts, these services risk large-scale automated junk submissions that waste service resources and consume users' attention or tokens. In this paper, we investigate a novel question for today's Web: "How can a Web service obtain evidence that a client is an agent?" We introduce the concept of "Proof-of-Agentness" (PoA), a mechanism for distinguishing agents from other automated clients, motivated by the parallel between human attention and an agent's inference resources. We outline two approaches: active verification through challenge-response and passive analysis of service-visible signals. Eight language models achieve an overall accuracy of 96.5% on our challenges, with an average overhead of 1,647 tokens. For passive analysis, we conduct an IRB-approved user study comparing the Web interactions of twelve human participants and representative Bot and Agent implementations, achieving an average classification accuracy of 95.8% across four classifiers, motivating further study.

0.29

A Large-Scale Evaluation of Cookie Notice Privacy and Accessibility for Blind and Low Vision Users

cs.CR ·2026-10-31 ·James M. Clarke, Maryam Mehrnezhad
similarity 0.62
Abstract

We present the first large-scale privacy and accessibility evaluation of cookie notices across 2,061 domains for blind and low vision (BLV) users. Using a Playwright-based crawler, a custom NVDA screen reader add-on, and the Lighthouse and WAVE accessibility tools, we measure UK GDPR/PECR compliance, tracker activity across pre-interaction, post-accept, and post-reject phases, and cookie notice accessibility for screen reader users. We find that 51 % of sites display a cookie notice, of which (at least) 50 % violate UK GDPR/PECR requirements. Accepting a notice produces a mean five-fold increase in tracker activity, and degrades page landmark structure on one in five sites. While 95 % of notices are readable by a screen reader, only 37 % carry a meaningful title, 39 % are announced immediately on page load, and 42 % use descriptive button labels. We reveal a systemic correlation: cookie notices failing legal privacy requirements disproportionately exhibit the weakest screen reader accessibility. This compounding privacy burden on BLV users demands urgent intervention from developers, regulators, and standards bodies.

0.31

When 200 OK Is Not an Answer: Silent Fetch Failures in Agentic Web Content Acquisition

cs.IR cs.CLcs.NI ·2026-10-31 ·Shuyang Zhang
positionhot direction similarity 0.61
Abstract · 10 pages, 3 figures

Web agents and answer engines fetch pages at question time, and liveness checks of cited sources treat an HTTP 200 response as success. We measure how often that assumption fails and what it does to measurements built on fetched pages. We fetched 10,307 URLs in three days with a polite static client and a rendered browser client: pages from 1,507 sites in five strata, random non-existent paths on every site, and the 4,795 URLs cited by expert-verified claims in ExpertQA. We define a taxonomy of what a 200 response contains, label the static 200 responses automatically, and check the labels against 300 items reviewed by an LLM-based assistant; a human check of 60 of them retained every reviewed usability label after discussion. By the automatic labels, 20.9% of the pages that a static client receives with status 200 do not contain the page's main content: app shells, bot challenges, block pages, error and placeholder pages, and partial content. On the held-out test split, the reviewed sample lowers this estimate from 25.5% to 20.0%. 13.1% of the sites whose random-path probe was answered returned 200. The error propagates: for the cited sources, treating 200 as content overstates the share of claim-source pairs that can be checked against live content by 12.2 points and inflates the apparent silent citation-drift rate from 5.8% to 21.9%. Cheap heuristics catch most of it. Text length, probe comparison and redirect-to-root, combined and tuned on held-out sites, recover 86% of unusable reviewed responses at a false-alarm rate near 5%. Content-reading detectors perform comparably: the best, the text classifier, gains five to seven points of recall at more than twice the false-alarm rate. We argue that agents and their evaluations should report fetch outcomes explicitly instead of inferring them from the status line and will release the labelled data and code.

0.36

AgentHeal: Evaluating Goal-Preserving Resource Recovery of Web Agents under Access Barriers

cs.CR cs.CL ·2026-10-29 ·Zhiyi Liu, Haohua Du, Haoqi Sun, Haisheng Tan
benchmarkcode/data similarity 0.61
Abstract · Code and Dataset are available at https://github.com/Lzzzzzzzzzzy/AgentHeal

Large language model (LLM) agents have demonstrated strong capabilities in automating complex information-seeking and transactional tasks across the open web. However, their real-world reliability remains limited by access barriers, such as dead links, authentication walls, rate limits, and anti-scraping defenses. These barriers block the primary execution path, causing agents to abort prematurely, loop indefinitely, or quietly compromise the user's original constraints. Existing evaluations either implicitly assume that target resources are freely accessible or focus on transient perturbations that simple retries can resolve, failing to capture scenarios where agents must actively reroute around impassable bottlenecks. To bridge this gap, this paper introduces AgentHeal, a comprehensive benchmark evaluating goal-preserving resource recovery, defined as the ability of web agents to detect non-transient access barriers, pivot to legitimate alternative sources, and fulfill the user's exact original objective. Spanning nine barrier families across a counterfactual sandbox and real-website replay, our experiments on leading LLM backbones show that average task success drops sharply from 89.4% to 43.7% under access barriers. Further analysis reveals that agents frequently fall for deceptive look-alikes, fail to escalate to users when necessary, violate access policies, or waste computation on futile retries, providing actionable diagnostics for developing resilient web agents.

0.32