CS Paper Radar

Prefix Sharing Is a Sorting Problem

Rong He

cs.DS cs.CL ·submitted 2026-09-12 04:02 UTC · v2
theorycode/data

Abstract

LLM serving reuses KV cache by exact prefix match, so when a prompt is assembled from a set of reusable pieces -- retrieved passages, tool definitions, few-shot exemplars -- the order chosen for those pieces determines how much computation can be shared. Every deployed system fixes that order by a single global convention. We prove this is optimal only when requests contain at most two pieces, and asymptotically wrong in general. Our main result is a structure theorem: the minimum prefix-trie cost equals min_H sum_x w(x) t_x(H) over binary hierarchies H on the requests, where t_x(H) is the canonical decomposition size of the set of requests needing chunk x. Choosing chunk orders is therefore equivalent to choosing one hierarchy over requests. The identity yields an O(3^m) exact algorithm, identifies the two-chunk case as minimum vertex cover, and shows that on the leave-one-out family the optimum is the minimum external path length of a binary tree -- the merge-sort recursion -- so a global order pays Theta(n^2) against a true cost of Theta(n log n). Agglomerative clustering by common intersection is a tight 1/2-approximation for the achievable saving. On BM25 retrieval traces over three BEIR corpora the resulting layout reduces prefill by 17-36% against production RAG ordering, and the margin widens with retrieval depth as the theory predicts. Serving requests in the hierarchy's DFS order finally lets a cache holding one request's context attain the unbounded-cache optimum exactly, so cache capacity and reorder window act as substitutes.

Comment
16 pages. v2: Corrected the spanning-tree lower-bound implementation and retrieval summary statistics; clarified the scope of empirical optimality claims. The nine reported lower bounds and main savings results are unchanged. Code: https://github.com/Darrenus/prefix-sharing-is-sorting

Closest papers

Nearest neighbours by meaning across the whole index.

When Is Deletion Ordering Tractable? From Update Dynamics to Permutation Structure

cs.DS cs.AI ·2026-10-01 ·Xinyu Wang, Ziyu Zhao, Yixuan He, Xiaowen Chang Alex Smola
similarity 0.67
Abstract

Given a fixed set of pending deletion requests, retraining from scratch after each request is prohibitive, so a prescribed request-wise policy processes them sequentially. The resulting terminal model can depend on their order. Rather than prescribing an ordering rule, we study the permutation objective induced by the fixed policy and ask when it admits simpler structure. We identify two independent reductions: position additivity represents the objective by request--position costs, reducing optimization to assignment and, with a shared positional profile, sorting; suffix localization removes dependence on the distant prefix while retaining interactions among the surviving requests. Under shared affine updates, we characterize the quadratic interactions that obstruct additivity, prove the reductions' independence, and show that suffix-conditioned assignment improves the approximation rate from O(p^L) toO(p^(2L)). Experiments recover both structures in executed objectives. A controlled damped-Newton sweep shows that stronger contraction shifts the objective toward shorter, more suffix-specific dependence, while two full-network policies exhibit distinct positional and within-suffix structure. Structures identified from compact execution sets also predict unseen orders. These results frame deletion ordering as identifying the computational structure induced by the executed updates.

0.18

KVCompass: A Discovery Plane for Internet-Scale Prefix Cache Sharing in LLM Serving

cs.DC ·2026-10-16 ·Shiyi Ling, Fei Fang, Chen Qian
published similarity 0.64
Abstract · 26 pages. Accepted for publication in Proceedings of the ACM on Measurement and Analysis of Computing Systems (POMACS)

KV cache reuse is one of the most effective ways to reduce LLM inference cost, but today its benefits are largely confined to a single cluster. When a provider serves the same model across geographically distributed regions, each region may repeatedly prefill the same popular prefixes, wasting GPU time and increasing time-to-first-token (TTFT). Exploiting reuse across regions involves discovering which worker currently holds a reusable KV cache for each request's prefix. We show that this KV cache discovery problem is structurally harder than conventional object lookup. Unlike hash-placement mechanisms such as DHTs, where an object's location is a deterministic function of its key, KV caches are created on the workers that serve requests, and their holder sets change with subsequent requests and KV cache evictions. KV cache discovery therefore calls for decoupling KV cache identity from placement. We present KVCompass, a discovery plane for Internet-scale KV cache sharing. The discovery index supports variable-length prefix lookup with constant-time local table reads at the edge, allowing each request to be forwarded to the worker that best balances KV cache reuse against load and network distance. Evaluation includes a 64-worker geo-distributed simulation and a 14-worker three-continent real-machine testbed. Evaluation shows that KVCompass achieves $1.3-14\times$ lower P99 TTFT than the baselines, up to $3.7\times$ lower P99 end-to-end latency, up to $4.7\times$ the throughput, and a peak load factor within 2% of the Oracle floor.

0.33

KVStream: Adaptive Parallel Storage Management for KV Caching in Large-Scale LLM Serving

cs.DC ·2026-10-22 ·Avinash Maurya, Jaiaid Mobin, Jie Ye, M. Mustafa Rafique, Franck Cappello +1 more
similarity 0.64
Abstract

Long-context LLM serving for agentic, coding, and document workloads reuses prefixes whose key-value (KV) cache outgrows GPU and host memory, putting storage I/O on the critical path of the inference step for retrieving prefixes. A single node-local NVMe absorbs all retrievals, while the parallel file system or cloud file store that nodes already mount carries almost no inference traffic. We cast prefix retrieval as aggregate-bandwidth scheduling across these tiers and present KVStream, a KV cache engine that batches a step's reads, defers writes behind the forward pass, replicates each chunk on every tier, and lets each tier adaptively claim the next unread chunks once it is free. Our performance model shows this routing attains the aggregate-bandwidth bound without prediction or tuning, whereas any static split misses it by a margin that grows as per-tier bandwidths drift or fluctuate. It also predicts that the time-to-first-token (TTFT) gain grows with load, as faster retrieval drains the queue, and disappears when host memory or arrival order keeps reuse in cache. Plugged into LMCache and evaluated on four models, six datasets, and up to 128 GPUs, KVStream retrieves within 5% of the bound, delivers up to 1.49$\times$ higher offline throughput than LMCache, and lowers mean TTFT by up to 6.1$\times$ under bursty arrivals.

0.06

Retrieval Sequencing in a Compact Storage System with Side Access

cs.DS cs.CC ·2026-10-29 ·Yağmur Gül
theory similarity 0.64
Abstract

In a compact storage system with side access, bins are arranged in adjacent vertical stacks. In each retrieval cycle, bins blocking access are lifted to create a temporary tunnel through which a robot extracts one requested bin from the side. The lifted bins are then lowered again, and the energy expenditure of the cycle equals the number of lifted bins. Given a pick list known in advance, we seek a retrieval order that minimizes total energy expenditure. For every number of stacks, we prove that the total energy separates into an instance constant and a pairwise penalty, and that every energy-optimal order is laminar within each stack. For two stacks, we prove that the retrieval sequencing problem is solvable in polynomial time. For three stacks, we give an exact polynomial-time reduction to a merge problem and characterize which merge-problem instances arise from the storage model. We define a recursive class of retrieval orders and construct a polynomial-time dynamic program that minimizes energy expenditure over this class. Finally, we quantify the additional energy savings from batching, which allows several requested bins to be retrieved in one cycle.

0.34

Comparison-Based Fair Division of Indivisible Chores

cs.GT ·2026-09-08 ·Zehan Lin, Shengxin Liu, Biaoshuai Tao, Shengwei Zhou
similarity 0.63
Abstract

We investigate the query complexity of fairly allocating $m$ indivisible chores among $n$ agents with additive cost functions. We depart from the standard cardinal model and assume only comparison access: an algorithm may ask an agent which of two bundles is less costly, but never observes numerical costs. Our first results concern proportionality up to one item (PROP1). We design comparison-based algorithms that compute PROP1 allocations using $O(n^3\log m)$ comparison queries. When the chores are arranged in a fixed order and allocations are required to be contiguous, we compute a contiguous PROP1 allocation using $O(n^3 \log^2 m)$ comparison queries. Our main result concerns the maximin share (MMS) guarantee. We show that for any fixed number of agents $n$ and constant $\varepsilon>0$, a $\left(13/11 +\varepsilon\right)$-MMS allocation can be computed with a comparison complexity logarithmic in $m$. Remarkably, comparison access suffices to match the state-of-the-art $13/11$ cardinal-access guarantee of Huang and Segal-Halevi up to an arbitrarily small loss. Furthermore, our result implies that the MMS distortion of comparison access (i.e., the worst-case multiplicative loss in MMS fairness incurred by observing only comparisons rather than numerical costs) is at most $13/11$. Finally, we show that, for three agents, an allocation satisfying envy-freeness up to one item (EF1) can be computed using $O(\log m)$ comparison queries.

0.16

Exact Memory-Time Optimization for Prefix-Cached Language Model Serving

cs.LG ·2026-10-02 ·Shivam Gupta
benchmarktheorycode/data similarity 0.62
Abstract · 16 pages, 5 figures. Code and reproducibility artifacts: https://github.com/shi1720/prefix-certificate-retention

Retaining language-model prefix states trades recomputation against storage time. Optimizing each cached block independently can overcount savings: a resident block is usable only when the required preceding prefix is also available. We introduce Prefix-Certificate Retention (PCR), an exact finite-trace formulation for static, grouped, reset-on-access timeouts. Usable-prefix rewards become nodes whose prerequisites are timeout thresholds and preceding hit certificates. The resulting maximum-weight closure reduces to one minimum cut, with graph size linear in the number of block lookups and timeout choices. A breakpoint theorem extends the construction to all nonnegative timeouts without discretization error. We also derive a linear-time-in-grid-size dynamic program for ordered timeouts and bounds that certify the cost of this restriction. Exhaustive small-instance checks and chronological replay of 39,632 public Mooncake requests validate the formulation. On the fixed grid, ordered timeouts attain the unrestricted training optimum in 118 of 120 trace-grouping-price cases. Heterogeneous retention improves several held-out memory-time tradeoffs, but finer training optimization does not uniformly improve transfer. The contribution is a tractable optimization model and an auditable benchmark for retention policies; the experiments measure usable prefix blocks and storage time, not GPU latency.

0.40

Dense Interprocedural Dominance in Acyclic Graphs: Context Bounds and Compact Queries

cs.DS ·2026-09-05 ·Peisen Yao
similarity 0.62
Abstract · We identified a substantive technical issue in the proof of the main theorem that affects several downstream results. Addressing the issue r…

Interprocedural dominance asks which program points every matched call-and-return execution must pass on its way to a target. An analysis that writes out immediate dominance pays for the transitive reduction of this order, the minimal dominator graph. Measured instances of that graph look near-linear, with edge-to-node ratios around 1.1--1.26~\cite{DeSutter2007}, but no program property is known to force this: it is unclear which structural restrictions imply sparse output, and whether dense output also rules out cheap queries. This paper answers both through calling contexts. A static node stands for the group of its reachable configurations; write $κ$ for the largest group size and $α$ for how many nodes have more than one configuration. One context per node keeps the tree with $n-1$ edges. Two already allow $Θ(n^2)$ cover edges in a sparse, loop-free, recursion-free ICFG with total degree three, constant-size procedures, and logarithmic stack depth, while the reachable configuration graph itself stays linear, so the blowup comes from grouping, not from many configurations. Three keep $Θ(n^2)$ covers even when the explicit call/return graph is acyclic: all target-reaching runs visit the mandatory procedures in the same order, and extra runs that stop short of any target remove the comparabilities that would otherwise kill the covers. Complementing the lower bounds, $α$ ambiguous nodes admit at most $\min\{\floor{n^2/4},(α+1)(n-1)\}$ cover edges, with $Ω(αn)$ attainable at stack depth one. And for every ICFG with at most two contexts per node, deleting a context group reduces dominance to dual-failure reachability, giving $O(n)$-word, $O(1)$-query representation after polynomial preprocessing. Explicit cover size, context ambiguity, and query-space cost are three separate quantities.

0.26

Beyond Distance Ordering: Resource Complexity and Universal Optimality of Exact Labeled Directed Shortest Paths

cs.DS cs.CC ·2026-09-04 ·Bin Cai
benchmarktheory similarity 0.61
Abstract · 45 pages, including appendices and references. One endpoint theorem is verified by an exact computer-assisted finite certificate

We study exact single-source shortest paths when the output is only the materialized labeled distance vector ($\mathrm{DIST}$), rather than a distance order. In the full deterministic comparison-addition model, the minimum worst-case number of additions on every fixed directed topology is exactly the maximum number $ρ_{\mathrm{fwd}}$ of forward nonsource endpoint classes over rooted vertex orders; the lower bound permits adaptive control, literals, and arbitrary mixed sums. This arithmetic law aligns with the comparison optimum on DAGs, where the full resource region is an exact rectangle. Cycles destroy that alignment: a two-spoke shared-hub graph has coordinatewise optima $(4,2)$ but requires five comparisons at the two-addition budget. Its $k$-spoke extension forces $k\log_2 k+O(k)$ comparisons at the addition optimum and has an entropy-tight deterministic tradeoff $C_{k+r}^*(H_k)=Θ(k+Λ_{k,r})$, where $Λ_{k,r}=\log_2(k!/[r!(r+1)^{k-r}])$, with leading constant one when $Λ_{k,r}/k\to\infty$. Because the two coordinatewise minima need not belong to one program, these conflicts lead to the same-program benchmark $\operatorname{OPT}_{\mathrm{DIST}}=\inf_A\sup_w(C_A(w)+P_A(w))$. An exact transcript-cone game yields one uniform interpreter whose charged addition-comparison cost equals $\operatorname{OPT}_{\mathrm{DIST}}$ on every topology; its optimal actions are synthesizable in polynomial space but may require exponential time. Finally, an active-core reduction and the current deterministic directed-SSSP bound give an efficient uniform $O\!\bigl(\operatorname{OPT}_{\mathrm{DIST}}\sqrt{\log(2+\operatorname{OPT}_{\mathrm{DIST}})\log\log(4+\operatorname{OPT}_{\mathrm{DIST}})}\bigr)$ charged-operation bound. Thus optimal numerical policies exist uniformly, while efficient constant-competitive navigation remains open.

0.28