CS Paper Radar

PonyTail: Profiling and Improving Service Tail Latency

Roee Wodislawski, Adam Morrison

cs.PF ·submitted 2026-10-03 15:57 UTC · v1

Abstract

Datacenter services must meet tight tail latency service-level objectives. Research on improving tail latency focuses on reducing queuing delay by approximating optimal request scheduling policies. As request scheduling nears optimality, the primary way to further improve tail latency is to reduce the service time of requests. Unfortunately, datacenter services lack obvious hotspots for general service time optimization. We point out a new optimization opportunity in targeting the tail service time of services with dispersive service times, which are common in datacenters. To show the feasibility and benefit of this approach, we present PonyTail, a performance analysis methodology and toolkit for analyzing service-level outlier behaviors associated with tail service time. PonyTail helps performance engineers identify control-flow and microarchitectural patterns responsible for service time outliers, as well as their root causes. We apply PonyTail to analyze five latency-critical services, including an in-memory database system and an online search engine. Based on PonyTail's analysis, we optimize some of the services, improving their 99th percentile service time and latency by 10.7%--46% and 21%--$5\times$, respectively.

Closest papers

Nearest neighbours by meaning across the whole index.

VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling

cs.AI ·2026-11-13 ·Ishaan Datta, Alind Khare, Koushik Kumar Dey, Ratnesh Jamidar, Anand Kumar Singh
similarity 0.67
Abstract

Interactive LLM services must meet latency service-level objectives (SLOs) for the time to first token (TTFT) and the time between tokens (TBT), as streaming and soft real-time applications require predictable token delivery. We identify decode jitters in both aggregated and disaggregated serving. A jitter delays every request in the batch, and long responses span hundreds of steps, so even rare overruns cause many SLO violations. In aggregated serving, SLO-aware schedulers set each step's prefill budget until its predicted latency just meets the TBT target, leaving no room for mispredictions or host stalls. Disaggregation meets average TBT targets at a cost in capacity, yet the steps that admit new requests still run late. We present \sysname, a scheduler that calibrates how much prefill work each decode step carries using the measured tail of its own iteration latency. The SLO determines how rare a late step must be, and \sysname leaves enough headroom in each iteration to prevent overruns and meet the target decode latency. This headroom slows prefill, so \sysname serves the shortest waiting prompt first. When TTFT deadlines grow with prompt length, the long prompts it delays have the most time to spare. Under disaggregation, it spaces admissions so that no request waits on more slow admission steps than its P99 target allows. On three attention families and varying workloads, \sysname sustains 2.4--2.9$\times$ the SLO goodput of the strongest baseline on Qwen3-8B and DeepSeek-V4-Flash, and at least 4.4$\times$ on DeepSeek-V2-Lite. With prefill and decode disaggregated on eight GPUs, \sysname meets the SLO for more than twice as many requests as state-of-the-art baselines.

0.19

Stochastic End-to-End Latency Modeling of the IoT-Edge-Cloud Continuum: Impact of Jitter and Traffic Variability on Deterministic Service Provisioning

cs.NI ·2026-08-26 ·Keyvan Aghababaiyan, Javier Gozalvez, Baldomero Coll-Perales
similarity 0.64
Abstract

6G will integrate communication and computing capabilities in a IoT-edge-cloud continuum, enabling nodes to distribute workloads across the continuum. To support time-sensitive services, both communications and computing latencies must be controlled. Two key sources of temporal variability are arrival-time jitter and traffic variability. They can both impact the timing at which data is generated, transmitted and processed, and the resulting fluctuations can propagate throughout the continuum, increasing latency uncertainty. This paper studies the impact of stochastic temporal variability on the ability to support end-to-end deterministic service levels across the continuum. To this end, we present a novel queueing-based end-to-end latency model for the continuum, which we openly release. The model jointly captures computing and communication latency, and characterizes the complete end-to-end latency distribution, including tail latency. Our analysis shows that services with stringent latency deadlines and larger computing demands are more sensitive to temporal variabilities, making local execution the preferred option. In contrast, services with more relaxed deadlines are more resilient to temporal variabilities when executed locally or at the edge despite higher average and tail latencies. Edge offloading is beneficial under good cellular connectivity and increasing local processing workloads, whereas cloud execution is more sensitive to traffic variabilities because of the additional communication latency. Our analysis also shows that services offloaded are more sensitive to traffic variability than jitter due to higher communication latencies. These findings highlight that effective service offloading must jointly consider service requirements and sources of temporal variability to guarantee deterministic service levels.

0.33

Tremulant: Latency in Serverless Platforms beyond the 99th Percentile

cs.DC ·2026-10-13 ·Linus Gustafsson, Trever Schirmer, David Bermbach
similarity 0.64
Abstract

Function-as-a-Service (FaaS) increases developer productivity by removing operational concerns such as managing hardware and software runtimes. Because function instances are started on demand, some requests must wait for a new instance, called a cold start. These are a common obstacle to more widespread adoption of FaaS, as they lead to unpredictable latency spikes that are not acceptable for some use cases. To solve this, many cold start optimizations have been implemented. However, comparatively little attention has been directed towards the slowest warm starts. With improvements in cold start times, slow warm starts emerge as a new problem for extreme tail events. By analyzing a dataset of several million requests across seven FaaS platforms, we show that the long tail of slow warm starts can be hundreds of times slower than their median performance, while a virtual machine we use as a baseline is substantially more consistent. We confirm this across four independent public datasets spanning 2020 to 2025, so the effect is neither new nor specific to one platform. Moreover, the warm share among the slowest calls starts to rise again at the highest percentiles: the most extreme latencies are not only cold starts, and cold-start mitigation alone cannot remove them.

0.18

Gutenberg: Taming Latency-Critical Cloud Services with Near-Data-Processing

cs.DC cs.AR ·2026-09-06 ·Qiushi Lin, Phillip B. Gibbons, Jovan Stojkovic, Yiwei Zhao
similarity 0.62
Abstract

Latency-critical cloud services place growing pressure on memory while requiring isolation, fairness, and predictable QoS. Near-data processing (NDP) reduces data movement by executing requests close to memory, and prior systems further improve locality through caching and replication. However, writes make replica maintenance expensive, while uneven compute and memory traffic can overload a few NDP units and increase tail latency. Existing throughput-oriented schedulers do not fully address these challenges for co-located cloud services. We present Gutenberg, a CPU+NDP for mutable, latency-critical cloud services. Gutenberg stages subpage updates in a CPU-resident delta buffer, allowing hot writable pages to remain replicated without eager full-page synchronization. It also adopts CPU helper cores to assist request execution when NDP execution or replica maintenance becomes costly. An online controller jointly decides page placement, replication, CPU/NDP execution, and routing using access patterns, queue pressure, and feedback from prior decisions. The system further enforces isolation and fair resource allocation across services. We also model-check CPU--NDP coordination protocol for correctness. We evaluate on TailBench using ZSim with Ramulator-calibrated memory timing. Across evaluated services, Gutenberg outperforms prior systems, reducing average and p99 latency by up to 80.4% and 85.8%. It also improves isolation and fairness while adapting to changing workload behaviors.

0.14

Characterizing Serverless Workloads: A Large-Scale Analysis of Function-as-a-Service Platforms

cs.DC cs.PF ·2026-11-03 ·Saman Akbari, Yassine Lazreg, Mohammadreza Malekabbasi, Manfred Hauswirth
code/data similarity 0.62
Abstract · Code available at: https://github.com/saman-akbari/characterizing-serverless-workloads

Serverless computing is a popular approach to application deployment without having to manage the underlying infrastructure. Its most common implementation is Function-as-a-Service (FaaS), in which applications are structured as fine-grained functions. FaaS providers face non-trivial tasks in the management of their platforms, including cold starts, auto-scaling, and scheduling. The design of optimized platforms requires an accurate understanding of realistic serverless workloads. However, existing research and platforms rely on insights from single, often outdated datasets. In this work, we conduct a large-scale, cross-platform comparative analysis of eight publicly available FaaS workload traces from major cloud providers (Alibaba, Azure, Huawei, IBM) and a scientific computing platform (Globus). We provide a comprehensive and up-to-date view of the serverless ecosystem by characterizing key dimensions such as invocation patterns, latency, cold start behavior, and resource utilization. Our analysis shows that although invocation patterns remain universal, execution times have shortened, shifting the performance bottleneck toward platform scheduling and dependency management. We also observe an architectural shift toward "Serverless 2.0" models, where concurrent, long-lived containers are increasingly utilized to bypass cold start penalties. Furthermore, we found resource over-provisioning and extreme skew in user request volumes. These findings provide actionable insights for designing more efficient FaaS platforms, informing improved approaches to targeted cold start mitigation, predictive resource management, and dependency-aware scheduling.

0.18

WaRP: Workload-Aware Request Planning for Vector Retrieval on Object Storage

cs.DB ·2026-10-21 ·Ruipeng Wan, Xinyu Zeng, Yiming Qiao, Huanchen Zhang
similarity 0.61
Abstract · 14 pages, 14 figures

Modern vector search systems keep payload vectors in cloud object storage and only a compact routing index on fast local storage, typically using cluster-based ANN indexes. This design inherits a problem existing systems do not address. On object storage, small Range-GETs have latency dominated by fixed per-request overhead and are billed per request. Existing readers either issue one request per probed cluster, or coalesce adjacent ranges with hand-tuned gap or span thresholds that do not transfer across datasets or query workloads. We formulate the grouping of probed-cluster byte intervals into object-store requests as a global per-query optimization problem and present WaRP, a workload-aware request planner that solves it efficiently. Given the sorted byte intervals, a limit on the number of requests per query, and an estimate of how many object-store requests from one query can proceed concurrently under the target serving configuration, WaRP searches for a merge plan that minimizes a concurrency-aware latency surrogate while satisfying the request limit. Using an optimized dynamic program and adaptive frontier search, WaRP returns a plan within $(1{+}ε)$ of the optimum, with per-query planning overhead under $10$ms even in the most demanding tested settings. A lightweight offline reordering pass further improves request mergeability without changing the semantics of ANN search. We implement WaRP inside Lance's I/O scheduler and evaluate it on AWS S3. We build IVFPQ indexes on seven datasets, including SIFT100M and DEEP100M at the 100M-vector scale. WaRP + Reorder improves QPS by $1.09--3.48\times$ and reduces GETs by $2.12--5.19\times$ relative to Lance, while preserving Lance's recall.

0.49

PANEM: A Heuristic Latency Model

cs.AR cs.PF ·2026-08-28 ·Lucas Crowthers, Mahesh Madhav
published similarity 0.60
Abstract · 12 pages, 23 figures, Presented at the 2026 IEEE International Symposium on Workload Characterization (IISWC 2026), Boulder, Colorado

Accurate pre-silicon memory modeling is essential for achieving meaningful representation of workloads on cloud-class many-core processors. Existing options force a poor tradeoff between fidelity and speed: fixed-latency models are fast but misleading, while cycle-accurate DRAM models are costly and difficult to scale across large study spaces or onto single-core environments. This paper presents PANEM, a lightweight event-driven heuristic model that has guided four generations of commercial server-core development at Ampere Computing. PANEM converts bandwidth-latency characterization data into a dynamic request-bytes/latency response, allowing miss latency to adapt to transient demand, queuing pressure, and read/write mix during simulation. Integrated into a single-core flow with configurable system-loading assumptions, PANEM enables realistic bandwidth constraints and contention-aware latency behavior without sacrificing throughput. Across a broad cloud workload trace suite, PANEM avoids the optimistic and pessimistic biases of fixed-latency baselines, yields more reliable conclusions for prefetching and dynamic throttling studies, and materially improves core-resource sizing decisions. These results show that a calibrated, contention-aware abstraction can deliver practical predictive value for industrial design-space exploration at simulation costs similar to fixed-latency models.

0.46

An Empirical Comparison of Monolithic and Microservices Architectures for an E-Commerce Application

cs.SE ·2026-08-16 ·Swayam Amol Katole
similarity 0.60
Abstract · 6 pages, 2 tables, conference paper submitted to IEEE i-COSTE 2026

Microservices architectures are widely adopted for their promised scalability and modularity, yet empirical evidence comparing their runtime performance to monolithic designs remains context-dependent. This paper presents an experimental comparison of a monolithic and a microservices implementation of the same e-commerce application, both backed by a shared PostgreSQL database. Using k6, we subject both systems to identical HTTP workloads at 50 and 100 virtual users (VUs) over 60-second runs, measuring throughput, latency, and error rates. At 50 VUs, both architectures perform similarly with no errors. At 100 VUs, the microservices design achieves 5.4% higher throughput, 25% lower average latency, and 39% lower p95 latency than the monolith, while exhibiting a lower median error rate (0.00% vs 0.69%). The monolith shows consistent order-creation failures under load, whereas microservices failures are transient and confined to the cart service in one run. These results suggest that, in this deployment context, decomposing the system into microservices improves scalability and tail latency under stress, while introducing distinct, service-specific failure modes that must be managed.

0.08