CS Paper Radar

An Alternative to Crawlers: Scalable P2P Network Monitoring Using the Network Scale-Up Method

Juan Marcos Ramirez, Sergio Díaz-Aranda, Leonhard Balduf, Antonio Fernández-Anta

cs.NI ·submitted 2026-11-16 16:18 UTC · v1

Abstract

\Ac{p2p} networks have remained an active area of research and development for decades. Originally used for data dissemination, they have later become instrumental to decentralized payment systems, blockchains, and Web3 applications. Since decentralization is a crucial property of these networks, they are continuously monitored to track attributes such as the geographical distribution of nodes, which serve as indicators of infrastructure concentration and resilience. However, monitoring typically relies on network crawls to derive accurate snapshots, which requires substantial computational and network resources. In contrast, \ac{nsum} is a statistical tool that estimates the prevalence of subpopulations based on responses to questions such as \enquote{How many people do you know in a specific group?} This work leverages the information embedded in Kademlia routing tables to develop a lightweight \ac{nsum}-based measurement and estimation framework that infers global network attribute distributions from a small set of nodes, offering an alternative to full-network crawls. As a case study, we evaluate our approach on \ac{ipfs}, a well-researched and widely used Web3 data storage and retrieval network. This is the first application of \ac{nsum} to decentralized network measurement. We evaluate our methodology on two datasets: the first comprises 360 full-network crawls collected over one month to assess our measurement-estimation approach across multiple simulated measurement scenarios, while the second dataset was collected through a deployment of the proposed lightweight measurement framework during May 2026. The results show that local Kademlia routing views, with only 10 nodes, preserve sufficient information to accurately estimate global attribute distributions, achieving normalized biases below \SI{10}{\percent} and closely tracking temporal changes in network composition.

Comment
10 pages, 5 figures

Closest papers

Nearest neighbours by meaning across the whole index.

How Bitcoin Forms Its Network: Peer-Table Sampling and Structural Properties

cs.NI cs.SI ·2026-09-07 ·Taki E. M. Abedesselam, Antonio Cruciani, Fabio Giacomelli, Lucianna Kiffer, Francesco Pasquale
similarity 0.72
Abstract

The global structure of P2P networks underlying modern cryptocurrencies is hidden by design: each node only knows its neighbors and maintains a local \textit{peer table} of IP addresses. The peer table of a node can be seen as the node's ``local view'' of the set of nodes currently in the network: it is constantly updated with information collected from the neighbors and it is used by the node to establish new connections when needed. The maintenance rules of the nodes' peer tables determine the global structure of the network and its evolution. Even though the global structure is unknown to any node of the network as well as to any external observer, the rules for the exchange of information between neighbors are compatible with the design and use of network \textit{crawlers} that can query the nodes and extract some information about their peer tables. In this paper we first analyze the data we crawled from nodes of the Bitcoin and Dogecoin P2P networks, to collect information about the distribution of IP addresses in the peer tables and to estimate the evolution of network size and churn rate; we then use the estimates to set up the parameters for a simulation of the Bitcoin network and we analyze the evolution of the network structure that we get from the simulation. Overall, our results show that Bitcoin's peer-table maintenance rules induce non-uniform, heavy-tailed visibility patterns while nevertheless generating a well-connected and structurally robust network.

0.27

You Get What You Sample: Evaluating Sampling Strategies for Web Security Measurements

cs.CR ·2026-09-10 ·Xuenan Zhang, Yuqing Yang, Giancarlo Pellegrino
top venue similarity 0.54
Abstract

Web measurement studies rely on domain datasets such as Tranco to quantify the prevalence and impact of security issues at scale, but exhaustively analyzing these datasets is often infeasible because of the cost of advanced analysis techniques, requiring the use of sampling. Despite its widespread use, sampling remains largely guided by convention---most commonly \emph{Top $N$} domain selection---rather than evidence, and its influence on the validity and generalizability of security findings has received little systematic evaluation. Consequently, it remains unclear whether common sampling strategies introduce systematic bias, distort observed vulnerability rates, or limit comparability across studies. In this work, we undertake, to the best of our knowledge, the first comprehensive investigation into how sampling methodologies affect the measurements and the conclusions. Through a comprehensive literature review and large-scale measurements of 500k Tranco and 24.8M Common Crawl hosts, we perform a comparative evaluation of datasets and sampling strategies. We show that, while Top $N$ sampling may be a rational strategy, the researchers have to bear in mind that Top $N$ does not reflect the overall distribution of the web. Instead, probability-based strategies yield stable, unbiased estimates for prevalence and many impact objectives. Hybrid sampling provides no advantages over pure probability sampling, as its deterministic prefix consistently contributes negatively to accuracy. Building on these results, we provide data-backed guidance for future studies, proposing to use an adaptive probability-based sampling strategy that remains effective even when the prevalence of the target issue is unknown.

0.43

IPv6 Hitlist Service: Lessons Learned From 10 Years of Operation

cs.NI ·2026-09-10 ·Oliver Gasser, Lion Steger, Patrick Sattler, Johannes Zirngibl
top venue similarity 0.54
Abstract · ACM SIGCOMM Computer Communication Review, Volume 56, Issue 3, October 2026

After becoming an Internet Draft more than 30 years ago, IPv6 has seen an increase in deployment and use in the past years. As measurements in the IPv6 Internet require new approaches due to the vastly larger address space, hitlists have come along as one possible source for finding IPv6 targets. One of the most prominent hitlists is provided by the IPv6 Hitlist Service. In this paper, we share insights from 10 years of operations of the IPv6 Hitlist Service: We show different evolutions of the service, highlighting important changes along the way. To better understand the representativeness of the hitlist, we perform a coverage analysis using real-world traffic data from a major central European ISP and Tier-1 network, finding that at least one address is known to the IPv6 Hitlist Service for 87.1 % of ASes and 56.5 % of /48 prefixes originating IPv6 traffic. We also share results from a conducted user survey and analyze users accessing the IPv6 Hitlist Service, finding diverse use cases and access patterns across time (e.g., one-off vs. continuous downloads) and available data (e.g., all vs. responsive addresses). Finally, we provide best practice recommendations when working with the hitlist and share lessons learned during its 10-year operation.

0.54

Locating and Enumerating Anycast: a Comparison of Two Approaches

cs.NI ·2026-09-18 ·Remi Hendriks, Tim Betzer, Ben Du, Raffaele Sommese, Mattijs Jonker +1 more
similarity 0.54
Abstract · 7 pages, 4 figures, 3 tables, ANRW'25

Anycast allows for providing services from multiple, geographically distant Points of Presence (PoPs), using a single IP address, to, e.g., improve resilience. Due to its opaqueness, it is often unknown which addresses are provisioned using anycast and, if so, where the PoPs are located. As anycast is widely used for critical Internet infrastructures (e.g., the DNS) efforts have been made to map anycast deployments. The current state-of-the-art mapping technique, iGreedy, relies on latency-based measurements, and is adversely affected by noise caused by, e.g., network processing delays. Previous work has shown that traceroute can alternatively be used to detect anycast. As traceroute reveals the hops a packet traverses, it may also be used to locate sites using geolocation data for hops near the anycast PoPs. This paper is the first to assess the performance of the traceroute-based approach at scale, by targeting 14k prefixes from an anycast census. Using ground truth we show traceroute achieves a slight increase in enumeration and geolocation precision over iGreedy. However, it suffers from overestimating the number of PoPs and incurs a 4 times increase in probing cost, making it unattractive for anycast censuses.

0.58

Host-Aware Web Crawling for LLM Pretraining Data

cs.CL ·2026-10-09 ·Malte Ostendorff
similarity 0.51
Abstract · Accepted at WaC-13 @ EMNLP 2026

Web crawling is a major source of large language model (LLM) pretraining data, yet data-selection pipelines discard a large share of crawled pages. We investigate how a crawler should allocate its budget across and within hosts to collect more high-quality pretraining data. Combining document-level quality classifier scores, web-graph centrality metrics, and Common Crawl's per-host crawl coverage, we find that roughly half of the variation in classifier scores occurs between hosts. In the crawls we analyze, a few pages from a host also provide a useful estimate of the quality of its remaining pages. Common Crawl uses harmonic centrality, a graph centrality metric, to guide its crawl allocation. In our simulations, using centrality to choose which hosts to crawl yields limited gains over random ordering and performs worse for some classifiers. For deciding how deeply to crawl each host, sampling a few pages and then fetching more from hosts with higher-scoring samples collects several times more high-quality pages per fetch than uniform depth or centrality-based allocation. Finally, Common Crawl's observed per-host coverage has little correlation with the evaluated quality signals. We estimate that tens of millions of high-quality pages remain among known but uncrawled URLs in each crawl examined, with most concentrated on a small share of hosts.

0.16

ProdCrawl: A Replayable Test Collection for Focused Crawling

cs.IR ·2026-10-29 ·Manuel Rey-Escobar, David Otero, Javier Parapar
code/datahot direction similarity 0.51
Abstract

A focused crawler searches for web pages on a specific topic under a constrained download budget. Since it can only fetch a limited number of pages, its success depends on choosing which links to follow. Focused crawlers are usually evaluated on live websites by harvest rate, the fraction of downloaded pages that are relevant. This has two problems: on the one hand, websites change over time, so a crawl cannot be repeated under the same conditions; on the other hand, there is no ground truth for which pages are relevant. As a result, comparisons between crawlers are unreliable. We present ProdCrawl, a replayable test collection for focused crawling. It comprises static mirrors of 621 company and organization websites, about 1.1 million pages, with each page labeled as a product page or non-product page based on a majority vote among three large language models. Any crawler can be run on exactly the same sites and scored against the same labels. We use ProdCrawl to compare simple baselines (breadth-first, depth-first and text classifiers) with an approach that scores each link by searching for similar link texts whose target pages have already been labeled. Our results show that text classifiers and keyword matching do not significantly outperform breadth-first crawling, whereas neural embeddings, which compare link texts by semantic meaning, outperform all simple strategies. Because every run evaluates over identical pages and labels, these results are exactly reproducible. The collection, codebase, and models are publicly released to support experimental replication and fair crawler comparison.

0.39

Improved Methods for k-core Community Search

cs.SI ·2026-09-15 ·Ian Chen, Haotian Yi, Arun Sharma, George Chacko, Tandy Warnow
benchmarkcode/data similarity 0.51
Abstract · 12 pages, 4 figures, submitted to CompleNet 2027

Community search based on user-specified query nodes is complementary to community finding or graph clustering. Prior work in community search is divided into optimizing for external separate- ness or internal cohesiveness, which does not scale well networks of over a billion edges. We present SteinerKCore, a new scalable k-core based community search algorithm for multi-vertex queries. We also present Par-ShellStruct, a parallel algorithm for building the ShellStruct data structure used for k-core community search. We show that our implemen- tations in Icebug, an open-source toolkit for large-scale network analysis, are both more efficient and more scalable than comparative tools, being able to perform on a benchmark network of 273M and 5.1B edges using just 64GB RAM and under 4 hours runtime with 16 CPUs.

0.46

Link prediction in complex networks via fusing node centrality and local similarity indices: a fair-protocol reassessment and parameter design principles

cs.SI ·2026-09-09 ·Yingying Zhang, Chengye Zhao
benchmark similarity 0.50
Abstract · 34 pages, 13 tables, 6 figures, 50 references

We reassess a published fusion method under a fair protocol. Local similarity indices are cheap but score every node pair without common neighbors zero, and weighting them by a node centrality product, as proposed for PageRank, is not robust once the protocol is made comparable across indices; on our eight-network panel the reported improvement does not survive that protocol for six of the seven local indices. We study these issues in a unified piecewise framework that modulates the local score multiplicatively by a centrality product when local information suffices and uses a small-dose centrality product as a completion term when it does not. A scale analysis and a reassessment on eight real-world networks (five-fold cross-validation, exhaustive negative samples) trace this to the O(n^-2) scale of the PageRank product, whose modulation factor is then a near-identity map, and to its rank correlation with degree exceeding 0.89; control arms show scale calibration is necessary but not sufficient. The min-max normalized DomiRank product instead has an O(1) range and substantial non-degree content, and a grid search yields unified parameters (omega=5, c=0.1) under which DR-JC/HPI/SO significantly outperform their local baselines already at the dataset level (p=0.027 and 0.035), the remaining four at the fold level only; DR-RA attains an average AUC of 0.9257, above nine of ten comparison methods, whereas degree-only preferential attachment reaches only 0.7736. Mechanism decomposition shows the two mechanisms contribute differently across index families and should be dosed separately, yielding a safety boundary 0 < c <= 0.3, a plateau omega in [0.5, 12] and a near-critical plateau sigma in [0.7, 0.95]. Two negative results bound the claim: under a degree-corrected benchmark the gains over local baselines vanish and reverse in sign, and the AUC gains do not translate into top-L hit rate.

0.16