CS Paper Radar

When Is Deletion Ordering Tractable? From Update Dynamics to Permutation Structure

Xinyu Wang, Ziyu Zhao, Yixuan He, Xiaowen Chang Alex Smola

cs.DS cs.AI ·submitted 2026-10-01 06:27 UTC · v1

Abstract

Given a fixed set of pending deletion requests, retraining from scratch after each request is prohibitive, so a prescribed request-wise policy processes them sequentially. The resulting terminal model can depend on their order. Rather than prescribing an ordering rule, we study the permutation objective induced by the fixed policy and ask when it admits simpler structure. We identify two independent reductions: position additivity represents the objective by request--position costs, reducing optimization to assignment and, with a shared positional profile, sorting; suffix localization removes dependence on the distant prefix while retaining interactions among the surviving requests. Under shared affine updates, we characterize the quadratic interactions that obstruct additivity, prove the reductions' independence, and show that suffix-conditioned assignment improves the approximation rate from O(p^L) toO(p^(2L)). Experiments recover both structures in executed objectives. A controlled damped-Newton sweep shows that stronger contraction shifts the objective toward shorter, more suffix-specific dependence, while two full-network policies exhibit distinct positional and within-suffix structure. Structures identified from compact execution sets also predict unseen orders. These results frame deletion ordering as identifying the computational structure induced by the executed updates.

Closest papers

Nearest neighbours by meaning across the whole index.

Prefix Sharing Is a Sorting Problem

cs.DS cs.CL ·2026-09-12 ·Rong He
theorycode/data similarity 0.67
Abstract · 16 pages. v2: Corrected the spanning-tree lower-bound implementation and retrieval summary statistics; clarified the scope of empirical opti…

LLM serving reuses KV cache by exact prefix match, so when a prompt is assembled from a set of reusable pieces -- retrieved passages, tool definitions, few-shot exemplars -- the order chosen for those pieces determines how much computation can be shared. Every deployed system fixes that order by a single global convention. We prove this is optimal only when requests contain at most two pieces, and asymptotically wrong in general. Our main result is a structure theorem: the minimum prefix-trie cost equals min_H sum_x w(x) t_x(H) over binary hierarchies H on the requests, where t_x(H) is the canonical decomposition size of the set of requests needing chunk x. Choosing chunk orders is therefore equivalent to choosing one hierarchy over requests. The identity yields an O(3^m) exact algorithm, identifies the two-chunk case as minimum vertex cover, and shows that on the leave-one-out family the optimum is the minimum external path length of a binary tree -- the merge-sort recursion -- so a global order pays Theta(n^2) against a true cost of Theta(n log n). Agglomerative clustering by common intersection is a tight 1/2-approximation for the achievable saving. On BM25 retrieval traces over three BEIR corpora the resulting layout reduces prefill by 17-36% against production RAG ordering, and the margin widens with retrieval depth as the theory predicts. Serving requests in the hierarchy's DFS order finally lets a cache holding one request's context attain the unbounded-cache optimum exactly, so cache capacity and reorder window act as substitutes.

0.28

Request Order Matters: Cache-History Sensitivity in Selective KV-Cache Reuse for Rolling Agents

cs.AI cs.LGcs.PF ·2026-10-05 ·Tiffany Gu, Annie Guan, Manshu Huang, Nitin Rao, Siddhant Shah +1 more
similarity 0.58
Abstract · 10 pages, 5 figures

Long-running agents repeatedly call an LLM while retaining most of their document window, evicting old documents, and appending new ones. These rolling updates break exact prefix caching and motivate non-prefix KV-cache reuse with selective recomputation. We show that persistent KV-cache reuse with selective recomputation can be history-dependent: in our rolling-agent workload, an unchanged prompt can produce different answers depending on the requests processed before it. At a matched 5% recomputation budget, document-aligned recomputation reduces answer variation across request orders from 69.0% with CacheBlend's token top-$k$ policy to 26.1%. When each prompt is evaluated after a different sequence of preceding requests, document-aligned recomputation improves fidelity to full prefill by 34.5-52.5 percentage points over token top-$k$, while both policies achieve approximately 5.7$\times$ median TTFT speedup. Our ablation study shows that, in our rolling-agent workload, contiguity is the main factor associated with robust selective recomputation.

0.18

When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reuse

cs.DC cs.LG ·2026-09-24 ·Yiyu Liu, Minlan Yu, Juncheng Yang
code/datahighly cited similarity 0.57
Abstract · 20 pages, 20 figures, 6 tables. Simulator: https://github.com/cacheMon/PrefixSim Traces: https://huggingface.co/datasets/harvardMadsys/2026_…

Long-running LLM applications and autonomous agents repeatedly send growing context, making prefix caching critical for reducing prefill cost. This raises a fundamental question: do decades of traditional cache-management techniques transfer directly to agentic and conversational LLM serving? We analyze production traces comprising over 20 billion processed tokens and evaluate 14 eviction algorithms across capacity-constrained HBM and large memory-pool settings. We find that simple Least Recently Used (LRU) is surprisingly robust, offering competitive performance against sophisticated traditional policies, though substantial headroom to Belady's optimum remains. The reason is structural: prefix reuse is dominated by the steady pacing of active sessions, making recency an unusually strong predictor for eviction. Nevertheless, prefix caching diverges from traditional workloads in ways that recency ignores, notably the prevalence of single-turn requests, extreme session footprint skew, and highly non-uniform miss costs because attention computation scales with token depth. To close the gap to optimality, we establish several design principles for prefix-cache management: keeping recency as the baseline while incorporating quick demotion to filter one-hit prompts, compute-aware eviction to prioritize expensive misses, and capacity-adaptive eviction granularity. Our traces and simulator are publicly available.

0.60

RR-Evict: Fine-Grained Prefix Cache Eviction beyond LRU for Agentic LLM Serving

cs.DC ·2026-09-26 ·Zaifeng Pan, Chris Wu, Zhengding Hu, Xinwei Qiang, Zhongkai Yu +1 more
similarity 0.57
Abstract

LLM-based agents execute long-horizon tasks through repeated model calls interleaved with tool execution and user interaction. As each call extends the history accumulated in previous turns, prefix caching avoids repeated prefill of the agent's entire context. However, the aggregate cache footprint grows with context length and concurrency, forcing serving systems to reclaim cached KV tensors. We identify recency synchronization: accesses to an agent's cached history refresh its cache nodes together, causing least recently used (LRU) eviction to concentrate on a few agents' private histories. When a fully evicted agent returns, it must recompute nearly its entire accumulated context, producing a large time-to-first-token (TTFT) outlier even if most other requests retain substantial cache reuse. We present RR-EVICT, a fine-grained prefix-cache eviction strategy that distributes reclamation across idle agent trajectories. RR-EVICT visits agents in round-robin order and evicts a tail chunk from each, preserving reusable prefixes for more agents when capacity remains for idle state. Returning agents can reuse these partial histories and recompute smaller missing suffixes. The policy requires no prediction of future arrivals or tool latency. We implement RR-EVICT in SGLang and evaluate conversational and coding-agent workloads under colocated and prefill-decode-disaggregated serving. Compared with LRU, RR-EVICT reduces P99 TTFT by up to 75.4% and P99 uncached prompt tokens by up to 65.7%.

0.44

KaleidoKV: Rearranging Evidence for Efficient KV Cache Reuse in Offline RAG Inference

cs.DB ·2026-11-14 ·Peiting Xie, Xingyu Tan, Xiangjun Zai, Chen Chen, Xiaoyang Wang +1 more
similarity 0.56
Abstract

Retrieval-augmented generation (RAG) augments large language models (LLMs) on question-answering (QA) tasks with retrieved evidence at inference time. Related questions in an offline batch often retrieve overlapping evidence, but prefix caching reuses key-value (KV) states only for identical token prefixes, making reuse dependent on evidence order. Although all questions and evidence are known before execution, existing reordering methods commit to early grouping decisions and can miss sharing opportunities that require several requests to move together. Evaluating alternative layouts can incur planning costs that offset inference savings, while concurrent requests may still recompute shared prefixes before their KV states become available. To address these challenges, we propose KaleidoKV, a framework that plans evidence orders across the batch and controls when requests enter the engine while preserving each request's evidence set and content. Joint layout planning revises formed groups through coordinated request moves and keeps a move only if it lowers planned prefill, the computation required when each shared prefix is computed once. A global prefix index selects evidence orders by locating reusable prefixes and evaluates each move incrementally from only the affected request paths. Prefix-ready admission tracks nested prefix dependencies, delaying requests whose shared prefixes are still being computed while allowing independent requests to proceed. Experiments on four QA workloads and four LLMs show that KaleidoKV achieves the highest throughput and lowest mean time to first token in all 16 settings, with approximately 1.24-1.75x the throughput of the strongest baseline, including planning time. With Llama-3.1-8B under the default configuration, actual prefill stays within 0.5% of planned prefill across all four workloads.

0.07

Tight Lower Bounds for Distributional Paging

cs.DS ·2026-10-29 ·Ali Vakilian
similarity 0.55
Abstract

Distributional paging seeks a polynomial-time eviction rule that competes with the optimal \emph{online} policy $\mathrm{OPT}$ when requests come from a known stochastic source. Two algorithms by Lund, Phillips, and Reingold [FOCS, 1994] are canonical. Their randomized \emph{dominating distribution} algorithm, for Markov sources, evicts a page drawn from a distribution over the cache satisfying pairwise linear inequalities; their deterministic \emph{median} algorithm, for sources with independent inter-arrival times across pages, evicts the cached page with the largest lower median of its conditional next-request time. Their original guarantees of $4$ and $5$ were recently improved to $2$ and $4$ by Pabbaraju and Vakilian [ICALP 2025]. We show that neither improved constant can be reduced in its respective source model. For Markov paging, we give two complementary factor-$2$ lower-bound constructions, each with an explicit polynomial-time dominating selector. A directed-cycle family has ratio $2k/(k+1)$, while a recurrent four-page family with cache size three has ratio $2-4\varepsilon/3$. For the independent inter-arrival model, we show that the median algorithm is not $c$-competitive for any $c<4$. These are separate results for distinct stochastic models. Matching the known upper bounds, they establish tight competitive ratios of $2$ and $4$, respectively, improving on previous lower bounds of $1.5907$ and $1.511$.

0.10

The Physical Cost of Context: Quality-Constrained Resource Accounting for Coding Agents

cs.SE cs.AI ·2026-10-26 ·Zhongming Yu, Yitong Ma, Hengjia Yu, Boqin Yuan, Xiangye Lin +2 more
publishedhot direction similarity 0.54
Abstract · Accepted at the NeurIPS 2026 Workshop on Resource-Aware Agentic AI (RAAAI). 15 pages, 2 figures, 6 tables

Coding agents resend a growing context on every model call, and context policies are usually judged by the prompt tokens they save. Executed work also depends on call count and prefix reuse: a caching runtime prefills only the tokens after the longest cached prefix. A shorter prompt can therefore cost more if it rewrites cached history. Our quality-gated protocol splits executed prefill into logical prompt minus reused prefix and measures KV residency, time, and energy. On 450 localization cases derived from SWE-rebench V2, preloading ten retrieved blocks (Eager) and adding anchored history compaction (Compact) both pass a five-point non-inferiority gate. In a separate 900-cell study, Eager cuts model calls by 35.7% and the GPU-energy proxy by 33.2% against on-demand search (Grep); Compact saves nothing further. Enabling prefix caching leaves Eager's prompt volume similar (+8.3%; its 95% CI spans zero) but cuts its uncached prefill by 67.7%. Deleting history can turn a token saving into extra work. With mini-SWE-agent on 100 SWE-bench Verified instances, keeping only the last five observations (Mask-5) cuts Eager's prompt volume by 33.3% but loses 135.9k reusable tokens per task. Uncached prefill rises by 608.5%, the energy proxy by 125.6%, and the policy misses the quality gate. Policies thus save physical work by shortening the trajectory and can multiply it by editing reusable history.

0.64

Ordered Policy Optimization

cs.LG cs.AIcs.ROstat.ML ·2026-11-06 ·Dongkyu Derek Cho, Pan Xu
top venue similarity 0.54
Abstract · 36 pages, 12 figures, 11 tables. Published in Proc. of the 40th Conference on Advances in Neural Information Processing Systems (NeurIPS 202…

Proximal Policy Optimization (PPO) stabilizes policy updates by clipping the importance ratio between the new policy and the data-collecting policy. The standard rule uses one fixed clipping range for all samples. While this choice offers simplicity and effectiveness, it fails to distinguish statistically harmful extreme ratio values from large, yet tolerable changes. We study how to adapt the clipping radius to the reliability of the induced importance ratio distribution. Our case study demonstrates that reliable updates require controlling these extreme ratios, rather than merely verifying that individual ratios fall within a fixed interval. Motivated by this observation, we propose Ordered Policy Optimization (\textsc{OPO}). In each minibatch, \textsc{OPO} sorts the importance ratios, selects the extreme ratios that potentially have the largest effect on the update, and assigns them adaptive sample-wise radii instead of using the fixed clipping radius. These radii are derived from a prescribed tail profile, which specifies desired target values for the selected extreme ratios. The resulting objective remains first-order and requires only minibatch sorting and a modified policy loss. Experiments conducted on MuJoCo and Atari benchmarks demonstrate that this simple modification achieves strong performance relative to established baselines.

0.34