CS Paper Radar

Caching Policies based on Object Sizes

Ridwanul Hasan Tanvir, George Kesidis

cs.PF ·submitted 2026-10-15 00:11 UTC · v1

Abstract

We evaluate eviction policies that consider the lengths of the objects being cached for different cache sizes (cost) and different workloads. Workloads include the Independent Reference Model and real-world production traces. In particular, hybrid policies are considered for segmented caches. A working-set approximation is developed for SIEVE as well as its length-aware extension.

Closest papers

Nearest neighbours by meaning across the whole index.

Performance and Cost-Aware Cache Provisioning

cs.PF ·2026-08-10 ·Ridwanul Tanvir, George Kesidis
similarity 0.67
Abstract

While traditional cache policy evaluations fix capacity - often at 0.1% of the dataset - and measure the resulting hit rate, practical edge-cloud deployments require balancing both storage and computational overhead as billed resources. Consequently, system operators frequently focus on a different objective: determining the minimum cache size needed to satisfy a specific Service-Level Objective (SLO) hit-rate. This paper explores this SLO-centric paradigm by analyzing the minimum capacity and execution time each policy requires to hit a defined target. Additionally, we show that dynamically adjusting the segment ratio in segmented policies based on historical workload patterns enhances efficiency. Through evaluations across real-world and synthetic traces, we present a novel hybrid segmented policy that reduces capacity requirements while keeping processing costs low.

0.04

Which Eviction Policy Should an LLM Cache Use? A Systematic Study Across Workloads, Capacities, and Encoders

cs.DB cs.LG ·2026-08-20 ·Yash Kulkarni, Shubham Harkare, Arvind Yogesh Suresh Babu
similarity 0.64
Abstract · 11 pages, 9 figures

Semantic caches reuse an LLM response when the incoming query embedding lies near a cached query, but proposed eviction policies have rarely been compared under one protocol. Using CLEVER, we evaluate FIFO, LRU, LFU, ARC, GDSF, a single-pass streaming adaptation of SISO, and a semantic-redundancy policy across three ordered, deduplicated query corpora, three cache capacities, and two encoders. No evaluated policy improves on LFU by more than 0.041 percentage points in any of the eighteen settings. Replacement is not irrelevant: FIFO and streaming SISO trail LFU by as much as 8.67 and 8.55 points, respectively, at tight capacity. We explain the missing upside with a conditional packing result. Under exact lookup and insert-on-miss, a newly inserted entry cannot have a resident neighbor within the hit radius, so a geometry-aware eviction rule receives little new redundancy signal. A separate audit exposes a larger problem with the evaluated operating point. At MiniLM's median nearest-neighbor threshold, only 2.1-3.9% of sampled LMSYS and QQP hits are judged answer-substitutable, reducing raw hit rates of 51-60% to quality-adjusted rates of 1.1-2.2%. The cross-encoder study further shows that thresholds do not transfer between embedding models. LFU is the strongest simple default in this protocol; deployment decisions should first establish answer validity and then test sub-point policy differences with exact search.

0.20

SR-Gadgets: Make Scan-Resistant Caching Practical

cs.PF ·2026-09-24 ·Yunjia Zheng, Juncheng Yang
hot direction similarity 0.63
Abstract

Block caches commonly serve scan-heavy I/O workloads, motivating extensive studies on scan-resistant eviction algorithms. Many of these algorithms adopt a multi-queue structure. However, they focus primarily on one-time scans and do not handle repeated scans well. Two important challenges from repeated scans are miss-ratio cliffs, where a small increase in cache size sharply reduces the miss ratio, and Belady's anomalies, where increasing the cache size increases the miss ratio. In this paper, we first develop two quantitative metrics to measure these behaviors. With these metrics, we find that LIRS is the only multi-queue algorithm that is scan-resistant (almost cliff- and anomaly-free). Contrary to conventional wisdom, we show that stack distance is not the secret sauce that makes LIRS scan-resistant. Instead, regulating the queues are the key to its scan resistance. Based on these insights, we design the \gadgetprefix Gadgets, easy-to-integrate augmentations that make existing algorithms scan-resistant without changing their eviction heuristics or queue structures. We implement the \gadgetprefix Gadgets in five algorithms: S3-FIFO, SIEVE, ARC, 2Q, and TinyLFU, and make them scan-resistant. Evaluated on 5,538 production traces, all augmented algorithms outperform their base versions, reducing miss ratios by up to 23.1% while consistently reducing cliffs and Belady's anomalies across the production traces.

0.31

The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems

cs.DC ·2026-09-23 ·Luchang Li, Shuaishuai Wang, Zhao Ruan, Dongfang Li, Bozhao Gong
code/data similarity 0.63
Abstract · 9 pages, 4 figures

Prefix caching is critical for efficient large language model (LLM) serving, particularly for agentic workloads that repeatedly invoke the model with a growing conversation and tool-use history. By reusing the key-value (KV) states of previously processed prefixes, prefix caching avoids redundant prefill computation. Its effectiveness, however, depends on retaining a sufficiently large set of KV cache states. Provisioning enough cache to preserve all historical KV states is prohibitively expensive and often unnecessary, whereas insufficient capacity can substantially degrade the cache hit rate. Determining the KV cache working set, defined as the minimum cache capacity required to achieve a target hit rate, is therefore essential for efficient cache provisioning and system design. We present KVSET, an online analyzer that estimates the KV cache working set of LLM serving workloads. KVSET uses the Mattson stack algorithm to efficiently estimate cache hit rates across a wide range of cache capacities. For each KV cache page, KVSET computes its LRU stack distance and compares it with the page number of each candidate capacity. This comparison determines whether the page would be a hit at each capacity without independently simulating every capacity configuration. KVSET therefore substantially reduces the computational and memory overhead of conventional capacity-by-capacity simulation and makes online working-set analysis practical. KVSET further determines the minimum cache capacity based on the maximum LRU depth among the prefix pages required to achieve the target hit rate. We validate KVSET using traces collected from production LLM workloads and show that its estimates closely match measurements from real cache deployments. The open-source implementation supports both online request processing and offline trace replay.

0.54

When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reuse

cs.DC cs.LG ·2026-09-24 ·Yiyu Liu, Minlan Yu, Juncheng Yang
code/datahighly cited similarity 0.63
Abstract · 20 pages, 20 figures, 6 tables. Simulator: https://github.com/cacheMon/PrefixSim Traces: https://huggingface.co/datasets/harvardMadsys/2026_…

Long-running LLM applications and autonomous agents repeatedly send growing context, making prefix caching critical for reducing prefill cost. This raises a fundamental question: do decades of traditional cache-management techniques transfer directly to agentic and conversational LLM serving? We analyze production traces comprising over 20 billion processed tokens and evaluate 14 eviction algorithms across capacity-constrained HBM and large memory-pool settings. We find that simple Least Recently Used (LRU) is surprisingly robust, offering competitive performance against sophisticated traditional policies, though substantial headroom to Belady's optimum remains. The reason is structural: prefix reuse is dominated by the steady pacing of active sessions, making recency an unusually strong predictor for eviction. Nevertheless, prefix caching diverges from traditional workloads in ways that recency ignores, notably the prevalence of single-turn requests, extreme session footprint skew, and highly non-uniform miss costs because attention computation scales with token depth. To close the gap to optimality, we establish several design principles for prefix-cache management: keeping recency as the baseline while incorporating quick demotion to filter one-hit prompts, compute-aware eviction to prioritize expensive misses, and capacity-adaptive eviction granularity. Our traces and simulator are publicly available.

0.60

CacheFit: The Rising Cost of Cache Replacement and Constraint-Driven Redesign of Last-Level Cache Replacement Policies

cs.AR cs.PF ·2026-10-05 ·Bita Aslrousta, Kaushal Mhapsekar, Darsh Asher, Azam Ghanbari, Joshua Kalyanapu +1 more
similarity 0.62
Abstract

The latency, power, and area costs of LLC replacement remain poorly explored not because simulators cannot be instrumented--event counters feeding a power model are routine--but because no cost model takes a replacement policy as input. CACTI and McPAT price regular structures, a cache or a core, not the small, oddly shaped arrays and dependent logic a policy is made of, and no event count reveals a victim-selection path's logic depth. Without a model, cost has never been an objective a policy search could optimize. In practice, a policy is first selected in simulation for its IPC, and its cost is reduced later in RTL, after the algorithm is fixed. Cost thus enters the design too late to shape it. This paper asks what happens when cost is present from the start. We build CacheFit, the first policy-design loop for LLC replacement with circuit-level cost inside the search objective. CacheFit connects ChampSim to HARCOM, a circuit-level cost model, and to a large language model that proposes policy variants. Every candidate is scored on latency, area, power, and speedup while its algorithm is still being written. The designer sets the area, power, and latency budgets, and every proposal and every lesson the search learns is stored as readable text that the designer can check and edit. We apply CacheFit to two published policies. From MPPPB, the most expensive baseline in power and transistors, it produces Fit-MPPPB: 60.3% less storage, 56.3% less static power, and 58.9% fewer transistors, with 1.1% higher IPC and a hit rate 7.5 percentage points higher on SPEC CPU2017. From SHiP++, the highest-IPC baseline, it produces Fit-SHiP++: 56% less storage, 61% less static power, and 56% fewer transistors, for 1.54% lower IPC and a hit rate 2.1 points lower. A policy designed with its cost in view can therefore be far cheaper to build at similar or better performance.

0.31

Heterogeneous Demand-Private Coded Caching: Exact Characterization for Two Users and Two Files

cs.IT ·2026-10-09 ·Tati V D N Eswar, Jithin Ravi
similarity 0.61
Abstract · 20pages

We consider the demand-private coded caching problem with heterogeneous file sizes and cache capacities, where each user must recover its requested file without learning any information about the demands of the other users. We first extend the virtual-user reduction technique to accommodate heterogeneous cache sizes, thereby enabling the construction of demand-private coded caching schemes from non-private schemes. Our main contribution is an exact characterization of the memory-rate trade-off for the fundamental setting of two users and two files with arbitrary file sizes and cache capacities. Specifically, we characterize the optimal trade-off through eight linear inequalities and establish achievability by constructing explicit coded caching schemes for the non-trivial corner points, followed by memory sharing. These constructions include novel asymmetric cache placement strategies tailored to heterogeneous file sizes and cache capacities. For the converse, we exploit the distributional equivalences induced by demand privacy to derive information-theoretic lower bounds that explicitly account for the underlying heterogeneity. Our results generalize the exact memory-rate trade-off characterization for the homogeneous demand-private coded caching setting with two users and two files, while quantifying the additional transmission cost imposed by demand privacy compared with the non-private heterogeneous setting.

0.15

Aker: Density-Aware Approximate Caching for Vector Search (Extended Version)

cs.DB ·2026-09-03 ·Sukjoon Oh, Minki Kang, Dohyun Kim, Baotong Lu, Jing Liu +3 more
top venue similarity 0.60
Abstract · Extended version of the paper published in Proceedings of the VLDB Endowment (PVLDB), Vol. 19, No. 10, pp. 2727-2740, 2026

Disk-based approximate nearest neighbor search (ANNS) incurs high I/O overhead due to frequent disk accesses during index traversal. Approximate caching, which reuses the results of past queries to serve future similar queries, offers a promising approach to bypass expensive disk searches. However, existing approaches suffer from two key limitations. First, their approximate hit predicates fail to simultaneously achieve high throughput and high accuracy, as they do not adapt to the varying local neighbor density in high-dimensional spaces. Second, they lack an effective refresh mechanism to maintain cache correctness under vector updates. We present Aker, an approximate cache for disk-based ANNS. Aker addresses these limitations through two core design choices. First, we introduce a per-query similarity threshold, where each cache entry maintains its own threshold that is dynamically adjusted based on observed cache hit patterns. This design enables Aker to adapt to neighborhood densities to preserve both efficiency and accuracy. Second, we propose del-consistency, a consistency model for ANNS caches that applies deletions eagerly and insertions lazily. Under this model, Aker implements a low-overhead refresh mechanism that bounds cache staleness and preserves high search accuracy. We integrate Aker into pgvector and evaluate it on representative workloads. Aker improves recall by up to 64 percentage points over prior solutions and increases QPS by up to 3.2x, while using 0.6x the memory of pgvector's shared buffers.

0.68