CS Paper Radar

Tremulant: Latency in Serverless Platforms beyond the 99th Percentile

Linus Gustafsson, Trever Schirmer, David Bermbach

cs.DC ·submitted 2026-10-13 15:17 UTC · v1

Abstract

Function-as-a-Service (FaaS) increases developer productivity by removing operational concerns such as managing hardware and software runtimes. Because function instances are started on demand, some requests must wait for a new instance, called a cold start. These are a common obstacle to more widespread adoption of FaaS, as they lead to unpredictable latency spikes that are not acceptable for some use cases. To solve this, many cold start optimizations have been implemented. However, comparatively little attention has been directed towards the slowest warm starts. With improvements in cold start times, slow warm starts emerge as a new problem for extreme tail events. By analyzing a dataset of several million requests across seven FaaS platforms, we show that the long tail of slow warm starts can be hundreds of times slower than their median performance, while a virtual machine we use as a baseline is substantially more consistent. We confirm this across four independent public datasets spanning 2020 to 2025, so the effect is neither new nor specific to one platform. Moreover, the warm share among the slowest calls starts to rise again at the highest percentiles: the most extreme latencies are not only cold starts, and cold-start mitigation alone cannot remove them.

Closest papers

Nearest neighbours by meaning across the whole index.

Characterizing Serverless Workloads: A Large-Scale Analysis of Function-as-a-Service Platforms

cs.DC cs.PF ·2026-11-03 ·Saman Akbari, Yassine Lazreg, Mohammadreza Malekabbasi, Manfred Hauswirth
code/data similarity 0.75
Abstract · Code available at: https://github.com/saman-akbari/characterizing-serverless-workloads

Serverless computing is a popular approach to application deployment without having to manage the underlying infrastructure. Its most common implementation is Function-as-a-Service (FaaS), in which applications are structured as fine-grained functions. FaaS providers face non-trivial tasks in the management of their platforms, including cold starts, auto-scaling, and scheduling. The design of optimized platforms requires an accurate understanding of realistic serverless workloads. However, existing research and platforms rely on insights from single, often outdated datasets. In this work, we conduct a large-scale, cross-platform comparative analysis of eight publicly available FaaS workload traces from major cloud providers (Alibaba, Azure, Huawei, IBM) and a scientific computing platform (Globus). We provide a comprehensive and up-to-date view of the serverless ecosystem by characterizing key dimensions such as invocation patterns, latency, cold start behavior, and resource utilization. Our analysis shows that although invocation patterns remain universal, execution times have shortened, shifting the performance bottleneck toward platform scheduling and dependency management. We also observe an architectural shift toward "Serverless 2.0" models, where concurrent, long-lived containers are increasingly utilized to bypass cold start penalties. Furthermore, we found resource over-provisioning and extreme skew in user request volumes. These findings provide actionable insights for designing more efficient FaaS platforms, informing improved approaches to targeted cold start mitigation, predictive resource management, and dependency-aware scheduling.

0.18

Fast Serverless LLM Cold Starts by Sharing and Prewarming Startup Work

cs.DC ·2026-11-12 ·Hongjin Kim, Igjae Kim, Kwanghoon Choi, Minho Ha, Jaehyuk Huh
similarity 0.66
Abstract · 10 pages, 10 figures, 3 tables

When a serverless LLM platform launches a model worker on demand, the cold start lasts roughly a minute, yet only 5-11% of it is spent loading weights. Most of the remainder goes to compilation, memory profiling and runtime setup, which each server performs again even though the outcome is identical across servers or independent of the model. We present Cairn. The server that is first to start a model publishes the weights and engine artifacts to a CXL memory pool shared across the cluster, and the remaining servers read them in place. Model-independent setup is completed ahead of time inside warm containers, and a template process replaces each container as soon as it is used. With ServerlessLLM on vLLM v0.20, six models from 1B to 14B, and two servers that share an emulated CXL pool, Cairn speeds up cold starts by 4.1-5.2x on servers without a checkpoint of their own and reduces median TTFT on a request stream by 64%.

0.31

Same Function, Different CPU: Quantifying Hardware Heterogeneity in Commercial FaaS Platforms

cs.DC ·2026-11-02 ·Jonas Heisterberg, Natalie Carl, Trever Schirmer, David Bermbach
similarity 0.64
Abstract

Serverless computing, particularly Function-as-a-Service (FaaS), abstracts operational concerns such as scaling, scheduling, and resource management from users. This abstraction, however, conceals a heterogeneous hardware fleet on which function execution environments are scheduled behind a unified function interface. Consequently, otherwise identical invocations may run on different CPU types that differ in performance, a hardware lottery that users can neither see nor control. Since the resulting variability is entangled with other sources, such as noisy neighbors or diurnal effects, its contribution to overall performance variability and how this contribution differs across providers remain unclear. In this paper, we systematically study hardware heterogeneity and its impact on function performance across four widely used commercial FaaS platforms, covering regional, configurational, and temporal differences. Using variance decomposition, we find that CPU heterogeneity alone explains 4 % to 97 % of the performance variance on AWS Lambda and Google Cloud Run Functions. On Azure Functions, in contrast, we identify a platform-level mechanism that produces bimodal execution durations and overshadows CPU effects. We observe that the CPU composition on AWS Lambda shifts within a single week. Additionally, we reveal that on Google Cloud Run Functions and Alibaba Function Compute, the availability of instruction set extensions such as SHA-NI differs between CPU types, widening the performance gap further. Since all platforms bill by duration, performance gaps between CPU types translate directly into cost, with invocations on the slowest CPU type costing up to three times as much as those on the fastest.

0.30

PonyTail: Profiling and Improving Service Tail Latency

cs.PF ·2026-10-03 ·Roee Wodislawski, Adam Morrison
similarity 0.64
Abstract

Datacenter services must meet tight tail latency service-level objectives. Research on improving tail latency focuses on reducing queuing delay by approximating optimal request scheduling policies. As request scheduling nears optimality, the primary way to further improve tail latency is to reduce the service time of requests. Unfortunately, datacenter services lack obvious hotspots for general service time optimization. We point out a new optimization opportunity in targeting the tail service time of services with dispersive service times, which are common in datacenters. To show the feasibility and benefit of this approach, we present PonyTail, a performance analysis methodology and toolkit for analyzing service-level outlier behaviors associated with tail service time. PonyTail helps performance engineers identify control-flow and microarchitectural patterns responsible for service time outliers, as well as their root causes. We apply PonyTail to analyze five latency-critical services, including an in-memory database system and an online search engine. Based on PonyTail's analysis, we optimize some of the services, improving their 99th percentile service time and latency by 10.7%--46% and 21%--$5\times$, respectively.

0.22

Reducing Cold-Start Latency in Serverless Applications via Dynamic Slicing

cs.DC cs.PLcs.SE ·2026-09-12 ·Georgios Alexopoulos, Konstantinos Karakatsanis, Nikolaos Alexopoulos, Dimitris Mitropoulos, Thodoris Sotiropoulos
similarity 0.63
Abstract

We present PyXtrim, a system that reduces the cold-start latency of serverless applications through debloating. We focus on Python, a dominant language for serverless applications whose dynamic features and extensive use of native extensions make traditional static debloating particularly challenging. PyXtrim frames debloating as a dynamic slicing problem, using the application's externally visible behavior as the slicing criterion. Everything outside the resulting slice is removed, both from the application and its dependencies. Our key technical contribution is that the slice is computed by a cross-language dynamic dependence engine that tracks data and control dependences across Python and native code and identifies operations that interact with the operating system, which form the slicing criterion. As a result, PyXtrim can effectively handle real-world applications that rely on dynamic features such as reflection, interoperate with native code and interact with system resources. Across 31 applications on AWS Lambda, PyXtrim reduces cold-start latency by 21.7% and peak memory usage by 17.1% at the median. This is more than double the reduction achieved by the state of the art, while debloating each application in minutes.

0.17

VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling

cs.AI ·2026-11-13 ·Ishaan Datta, Alind Khare, Koushik Kumar Dey, Ratnesh Jamidar, Anand Kumar Singh
similarity 0.63
Abstract

Interactive LLM services must meet latency service-level objectives (SLOs) for the time to first token (TTFT) and the time between tokens (TBT), as streaming and soft real-time applications require predictable token delivery. We identify decode jitters in both aggregated and disaggregated serving. A jitter delays every request in the batch, and long responses span hundreds of steps, so even rare overruns cause many SLO violations. In aggregated serving, SLO-aware schedulers set each step's prefill budget until its predicted latency just meets the TBT target, leaving no room for mispredictions or host stalls. Disaggregation meets average TBT targets at a cost in capacity, yet the steps that admit new requests still run late. We present \sysname, a scheduler that calibrates how much prefill work each decode step carries using the measured tail of its own iteration latency. The SLO determines how rare a late step must be, and \sysname leaves enough headroom in each iteration to prevent overruns and meet the target decode latency. This headroom slows prefill, so \sysname serves the shortest waiting prompt first. When TTFT deadlines grow with prompt length, the long prompts it delays have the most time to spare. Under disaggregation, it spaces admissions so that no request waits on more slow admission steps than its P99 target allows. On three attention families and varying workloads, \sysname sustains 2.4--2.9$\times$ the SLO goodput of the strongest baseline on Qwen3-8B and DeepSeek-V4-Flash, and at least 4.4$\times$ on DeepSeek-V2-Lite. With prefill and decode disaggregated on eight GPUs, \sysname meets the SLO for more than twice as many requests as state-of-the-art baselines.

0.19

GearUp: DAG-aware Acceleration for Serverless Inference Serving via Slack-guided Pre-loading

cs.DC cs.LGcs.OS ·2026-10-28 ·Qi Qi, Yifan Sui, Hanfei Yu, Hong Zhang, Hao Wang
similarity 0.61
Abstract

Serverless platforms increasingly host machine learning (ML) inference workflows, modeled as directed acyclic graphs (DAGs) of interdependent functions. Beyond container startup, each ML node must load libraries, model weights, and CUDA kernels before it can run. These costs accumulate along dependent DAG paths and place initialization on the end-to-end critical path. Existing systems optimize one half of this problem. DAG-agnostic pre-loading prepares nodes the request may never reach, wasting host and GPU memory. DAG-aware serving routes and scales whole functions, leaving the loading stages inside a node untouched. This paper presents GearUp, a serverless DAG inference serving system that removes ML-node loading from the end-to-end critical path at low resource cost. GearUp profiles the loading stages of each node and schedules them inside the slack time the DAG exposes, hiding initialization behind upstream execution. It reuses initialized instances within and across requests to avoid redundant cold starts. A probabilistic, traffic-adaptive keep-alive policy sets how long idle instances are retained, balancing memory cost against cold-start risk under bursty load. On the AMBER Alert and FINRA-ML workflows with NVIDIA L40S GPUs, GearUp reduces median end-to-end latency by 64.5% and 75.9% against SMIless, for median speedups of 3.1x and 10.0x over cold execution. Integrated GPU and host memory allocation fall by up to 83.9% and 94.6% against eager pre-loading in InstaInfer. GearUp achieves 4.2x strong scaling on four workers.

0.25

FaaSpot: Fault-Tolerant Serverless Workflow Execution on Spot Instances

cs.DC ·2026-10-19 ·Bandhan Patel, Sushruth Harsha, Kwangsung Oh, In Kee Kim
publishedcode/data similarity 0.60
Abstract · 14 pages, 12 figures, 7 tables. To appear in the 27th International Middleware Conference (Middleware '26), December 14-18, 2026, Tarragona,…

Serverless computing is increasingly used to run complex DAG workflows across diverse applications, and individual function execution times have grown to support them, pushing workflow makespans to hours. Open-source serverless platforms, deployed on a pool of VMs, typically use on-demand instances, and their cost increases with the makespan. Spot instances are far cheaper, but a preemption is likely before the workflow finishes. A preemption at a fan-out node or a task feeding a fan-in node stalls every parallel branch, cascading a single preemption into a workflow-wide failure. Prior systems address this challenge by migrating high-impact tasks to reliable, non-spot resources, sacrificing spot's cost advantage. We present FaaSpot, a serverless workflow execution system that runs every workflow task on spot instances with no on-demand fallback. Our key insight is that cascade-prone positions are determined by the DAG's topology and can be identified before execution. Therefore, FaaSpot analyzes the workflow DAG once at submission time and classifies each task by the impact of its preemption. Selective checkpointing, survival-aware scaling, and lineage-based retry then act on this single classification, with no runtime coordination. We evaluate FaaSpot on a trace-driven spot emulator with eight DAG workflows against four state-of-the-art baselines. FaaSpot completes 89.03% of runs at $0.018 per successful run, 21pp (percentage points) above the average baseline success rate and at a lower cost per success than every baseline, while reducing mean latency by 2.3x.

0.52