CS Paper Radar

You Get What You Sample: Evaluating Sampling Strategies for Web Security Measurements

Xuenan Zhang, Yuqing Yang, Giancarlo Pellegrino

cs.CR ·submitted 2026-09-10 08:21 UTC · v2
published

Abstract

Web measurement studies rely on domain datasets such as Tranco to quantify the prevalence and impact of security issues at scale, but exhaustively analyzing these datasets is often infeasible because of the cost of advanced analysis techniques, requiring the use of sampling. Despite its widespread use, sampling remains largely guided by convention---most commonly \emph{Top $N$} domain selection---rather than evidence, and its influence on the validity and generalizability of security findings has received little systematic evaluation. Consequently, it remains unclear whether common sampling strategies introduce systematic bias, distort observed vulnerability rates, or limit comparability across studies. In this work, we undertake, to the best of our knowledge, the first comprehensive investigation into how sampling methodologies affect the measurements and the conclusions. Through a comprehensive literature review and large-scale measurements of 500k Tranco and 24.8M Common Crawl hosts, we perform a comparative evaluation of datasets and sampling strategies. We show that, while Top $N$ sampling may be a rational strategy, the researchers have to bear in mind that Top $N$ does not reflect the overall distribution of the web. Instead, probability-based strategies yield stable, unbiased estimates for prevalence and many impact objectives. Hybrid sampling provides no advantages over pure probability sampling, as its deterministic prefix consistently contributes negatively to accuracy. Building on these results, we provide data-backed guidance for future studies, proposing to use an adaptive probability-based sampling strategy that remains effective even when the prevalence of the target issue is unknown.

DOI
10.1145/3830454.3846754

Closest papers

Nearest neighbours by meaning across the whole index.

PhiShark2026: A Multi-Layer Active-Web Raw-Evidence Dataset for Phishing Website Research

cs.CR ·2026-08-24 ·Furkan Çolhak, Ferhat Demirkıran, Hasan Dağ, Alexander Iliev
similarity 0.60
Abstract · 13 pages, 4 figures, 13 tables

Phishing websites are short-lived and rapidly changing, yet many phishing datasets reduce observations to URLs or precomputed features, constraining researchers to predefined representations and discarding the underlying evidence needed to derive alternative features, apply new extraction methods, examine cross-layer relationships, and reanalyze observations as phishing techniques evolve. This study addresses this limitation with a multi-layer active-web dataset comprising 67,502 scans, including 33,387 phishing observations from operational feeds and 34,115 screened benign reference observations. The corpus preserves raw evidence across HTML content and screenshots, URL and redirect behavior, HTTP and security headers, compliance files, TLS certificates, DNS and domain registration, open ports, geolocation and accessibility measurements, and network infrastructure, while explicitly recording unavailable evidence rather than treating it as negative observations. To avoid misleading infrastructure attribution on shared platforms, the study applies a hosting-aware evidence model that masks provider-owned infrastructure signals for free-hosted tenant pages while retaining meaningful page- and transport-level evidence. Characterization reveals systematic differences between phishing and benign websites across web-resource usage, domain maturity, mail and policy configuration, security headers, and infrastructure context. By preserving raw artifacts together with acquisition metadata and explicit evidence availability, the corpus provides an inspectable and reproducible foundation for future phishing measurement and dataset research.

0.30

Fetch Me If You Can: A Comprehensive Analysis of Fetch Metadata

cs.CR ·2026-10-11 ·Pedro Bernardo, Alex Ploner, Marco Squarcina, Mauro Tempesta, Matteo Maffei
similarity 0.58
Abstract

Effective server-side defenses against cross-site attacks depend on understanding the context in which browser requests are generated. However, servers often lack the information needed to distinguish legitimate requests from potentially malicious cross-site interactions. Fetch Metadata, a security mechanism introduced in 2019, addresses this limitation by adding a set of HTTP headers to outgoing browser requests. These headers provide servers with information about a request's context and provenance, enabling them to enforce request isolation policies accordingly. However, the extent of Fetch Metadata's real-world adoption and its interaction with existing security mechanisms remain largely unexplored. In this paper, we present the first large-scale measurement study of Fetch Metadata adoption. We analyze the top 1M websites in the Tranco list to detect deployments of Fetch Metadata isolation policies and identify custom policies used in the wild. We then examine how these policies address cross-site threats and assess their security coverage. Finally, we provide the first systematic comparison of Fetch Metadata with existing client-side security mechanisms, revealing security gaps that arise from differences between client-side and server-side enforcement.

0.36

Exploited Before It Was Fixed? Vulnerability Timelines from Open, Machine-Readable Evidence

cs.CR ·2026-11-17 ·Cédric Bonhomme, Alexandre Dulaunoy
similarity 0.57
Abstract · 8 pages

-05-X-03), Known Exploited Vulnerabilities (KEV) catalogs, and exploitation sightings. We introduce an evidence model implemented in Vulnerability-Lookup that selects the earliest evidence for each lifecycle event, distinguishes vulnerability-specific records, preserves date precision, and ensures traceability to original sources. Among 3,758 vulnerability identifiers with exploitation evidence, 98% have disclosure dates, while only 15.7% have patch evidence. Vendor-side dates required by VTEM are entirely absent. Most exploitation dates originate from catalog listings, including 1,041 entries associated with bulk catalog imports. After excluding these entries and ambiguous same-day observations, 59 vulnerabilities exhibit patch evidence following exploitation evidence, with a median interval of 7.6 days, predominantly between catalog listings and distributor fixes. Evidence traceability also exposes systematic measurement errors: one collector incorrectly classified exploit publications as active exploitation, accounting for 40% of initial exploitation dates, while another catalog assigned vulnerability publication timestamps to 11% of its listings. Our findings demonstrate that exploitation-to-patch intervals are computationally measurable, but their reliability depends on the availability, precision, and provenance of published evidence. We identify the additional disclosure, exploitation observation, and patch availability timestamps needed to improve vulnerability lifecycle measurements.

0.45

Escalate or Close: How Fast-Tier Models Handle Held-Out Web Attack Families in Two-Tier Triage

cs.CR ·2026-11-05 ·Maciej Kisielewicz, Michał Kędziora, Ngoc Thanh Nguyen
similarity 0.57
Abstract · 20 pages, 4 figures, 24 tables

In two-tier triage a fast model decides most web requests and escalates the rest to a slow tier, so an attack the fast tier closes is never examined. We hold each of seven attack families out of training, recalibration and thresholds, splice the payloads into parameters of real browsing traffic, and compare zero-shot decision models, which return a probability for a typed question, with supervised detectors, a local language model, the OWASP Core Rule Set and an anomaly baseline. Detectors trained on the known families can give much of a held-out family a confident benign verdict: over 10 training runs, ModernBERT gives 37% to 60% of held-out path traversal a probability below 0.05 and escalates only 50% to 81% of it within a 5% benign budget, against nearly all of the same requests when traversal is known. Jev and Clef, two hosted decision models whose budget decisions do not depend on the held-out family, escalate at least 97.6% of every family within the same budget, but their undisclosed training data may contain these public families. Tests on recent exploits need controls for request shape: Jev escalates 37% of benign twins (exploit requests with the attack replaced by legitimate values), and detectors trained on older exploits learn what scanner requests look like. A search guided by Jev's probability gets 43% of held-out attacks past Jev within 300 queries, but Jev's answers vary: asked again, it escalates those requests in 60% of its answers.

0.56

FlowAtom: Atom-Based Evidence Aggregation for Multi-Label Website Fingerprinting

cs.LG cs.NI ·2026-09-24 ·Chongru Fan, Wentao Huang, Wei Wang, Zhenquan Ding, Jinqiao Shi +2 more
code/data similarity 0.57
Abstract · 5 pages. Submitted to ICASSP 2027

Identifying the set of monitored websites in mixed encrypted traffic is challenging because an individual flow often provides only partial evidence of website identity. To address this challenge, we propose FlowAtom, which constructs shared prototypes, called Atoms, from flow representations without website labels. Specifically, FlowAtom pretrains a flow encoder on external unlabeled traffic and aggregates Atom responses across flows within each observation window into a fixed-dimensional, permutation-invariant representation for monitored website-set prediction. Across Direct HTTPS, Trojan, and VMess, FlowAtom achieves micro-F1 scores of 97.82%, 94.43%, and 93.92% in closed-world evaluation, respectively, and consistently outperforms the evaluated baselines in open-world evaluation on windows containing monitored visits. The code is available at https://github.com/aimafan123/FlowAtom.

0.34

Security of Deep Research Agents: Attack Surface, Defenses, Benchmarks, and Agenda Seeding against Research Planning

cs.CR cs.AI ·2026-11-15 ·Yang Luo, Guangpu Chen, Yichen Liu, Zhaoyu Chen, Yifeng Gao +3 more
survey similarity 0.56
Abstract · 31 pages, 5 figures

Deep research and agentic search agents autonomously retrieve content from the open web and generate reports with citations, and their outputs are often taken directly as research conclusions. This paper systematizes security research in this area along four stages, namely retrieval, planning, evidence integration, and report generation. We present a unified threat model, a taxonomy of attacks and defenses, and a review of evaluation benchmarks, and we align the area dimension by dimension with retrieval-augmented generation (RAG) poisoning, prompt injection, and generative engine optimization (GEO). Coding 34 primary studies shows that only 26.5% of them involve the planning stage, and none performs an end-to-end poisoning evaluation on commercial products. To address this gap, we propose the agenda seeding attack, in which the attacker writes no instructions and only introduces a fictitious evaluation metric. Across 3,520 sandboxed research runs under the same budget, GEO and prompt injection are nearly ineffective, whereas agenda seeding makes the attacker's target the top pick in 16% to 28% of reports. Injection detection and query anchoring fail against it, verification prompting hurts utility, and our proposed provenance tagging preserves utility but can be bypassed by multi-site bait. Finally, we propose eight research directions.

0.16

Host-Aware Web Crawling for LLM Pretraining Data

cs.CL ·2026-10-09 ·Malte Ostendorff
similarity 0.56
Abstract · Accepted at WaC-13 @ EMNLP 2026

Web crawling is a major source of large language model (LLM) pretraining data, yet data-selection pipelines discard a large share of crawled pages. We investigate how a crawler should allocate its budget across and within hosts to collect more high-quality pretraining data. Combining document-level quality classifier scores, web-graph centrality metrics, and Common Crawl's per-host crawl coverage, we find that roughly half of the variation in classifier scores occurs between hosts. In the crawls we analyze, a few pages from a host also provide a useful estimate of the quality of its remaining pages. Common Crawl uses harmonic centrality, a graph centrality metric, to guide its crawl allocation. In our simulations, using centrality to choose which hosts to crawl yields limited gains over random ordering and performs worse for some classifiers. For deciding how deeply to crawl each host, sampling a few pages and then fetching more from hosts with higher-scoring samples collects several times more high-quality pages per fetch than uniform depth or centrality-based allocation. Finally, Common Crawl's observed per-host coverage has little correlation with the evaluated quality signals. We estimate that tens of millions of high-quality pages remain among known but uncrawled URLs in each crawl examined, with most concentrated on a small share of hosts.

0.16

When 200 OK Is Not an Answer: Silent Fetch Failures in Agentic Web Content Acquisition

cs.IR cs.CLcs.NI ·2026-10-31 ·Shuyang Zhang
positionhot direction similarity 0.55
Abstract · 10 pages, 3 figures

Web agents and answer engines fetch pages at question time, and liveness checks of cited sources treat an HTTP 200 response as success. We measure how often that assumption fails and what it does to measurements built on fetched pages. We fetched 10,307 URLs in three days with a polite static client and a rendered browser client: pages from 1,507 sites in five strata, random non-existent paths on every site, and the 4,795 URLs cited by expert-verified claims in ExpertQA. We define a taxonomy of what a 200 response contains, label the static 200 responses automatically, and check the labels against 300 items reviewed by an LLM-based assistant; a human check of 60 of them retained every reviewed usability label after discussion. By the automatic labels, 20.9% of the pages that a static client receives with status 200 do not contain the page's main content: app shells, bot challenges, block pages, error and placeholder pages, and partial content. On the held-out test split, the reviewed sample lowers this estimate from 25.5% to 20.0%. 13.1% of the sites whose random-path probe was answered returned 200. The error propagates: for the cited sources, treating 200 as content overstates the share of claim-source pairs that can be checked against live content by 12.2 points and inflates the apparent silent citation-drift rate from 5.8% to 21.9%. Cheap heuristics catch most of it. Text length, probe comparison and redirect-to-root, combined and tuned on held-out sites, recover 86% of unusable reviewed responses at a false-alarm rate near 5%. Content-reading detectors perform comparably: the best, the text classifier, gains five to seven points of recall at more than twice the false-alarm rate. We argue that agents and their evaluations should report fetch outcomes explicitly instead of inferring them from the status line and will release the labelled data and code.

0.36