CS Paper Radar

Fast Serverless LLM Cold Starts by Sharing and Prewarming Startup Work

Hongjin Kim, Igjae Kim, Kwanghoon Choi, Minho Ha, Jaehyuk Huh

cs.DC ·submitted 2026-11-12 02:11 UTC · v1

Abstract

When a serverless LLM platform launches a model worker on demand, the cold start lasts roughly a minute, yet only 5-11% of it is spent loading weights. Most of the remainder goes to compilation, memory profiling and runtime setup, which each server performs again even though the outcome is identical across servers or independent of the model. We present Cairn. The server that is first to start a model publishes the weights and engine artifacts to a CXL memory pool shared across the cluster, and the remaining servers read them in place. Model-independent setup is completed ahead of time inside warm containers, and a template process replaces each container as soon as it is used. With ServerlessLLM on vLLM v0.20, six models from 1B to 14B, and two servers that share an emulated CXL pool, Cairn speeds up cold starts by 4.1-5.2x on servers without a checkpoint of their own and reduces median TTFT on a request stream by 64%.

Comment
10 pages, 10 figures, 3 tables

Closest papers

Nearest neighbours by meaning across the whole index.

GearUp: DAG-aware Acceleration for Serverless Inference Serving via Slack-guided Pre-loading

cs.DC cs.LGcs.OS ·2026-10-28 ·Qi Qi, Yifan Sui, Hanfei Yu, Hong Zhang, Hao Wang
similarity 0.66
Abstract

Serverless platforms increasingly host machine learning (ML) inference workflows, modeled as directed acyclic graphs (DAGs) of interdependent functions. Beyond container startup, each ML node must load libraries, model weights, and CUDA kernels before it can run. These costs accumulate along dependent DAG paths and place initialization on the end-to-end critical path. Existing systems optimize one half of this problem. DAG-agnostic pre-loading prepares nodes the request may never reach, wasting host and GPU memory. DAG-aware serving routes and scales whole functions, leaving the loading stages inside a node untouched. This paper presents GearUp, a serverless DAG inference serving system that removes ML-node loading from the end-to-end critical path at low resource cost. GearUp profiles the loading stages of each node and schedules them inside the slack time the DAG exposes, hiding initialization behind upstream execution. It reuses initialized instances within and across requests to avoid redundant cold starts. A probabilistic, traffic-adaptive keep-alive policy sets how long idle instances are retained, balancing memory cost against cold-start risk under bursty load. On the AMBER Alert and FINRA-ML workflows with NVIDIA L40S GPUs, GearUp reduces median end-to-end latency by 64.5% and 75.9% against SMIless, for median speedups of 3.1x and 10.0x over cold execution. Integrated GPU and host memory allocation fall by up to 83.9% and 94.6% against eager pre-loading in InstaInfer. GearUp achieves 4.2x strong scaling on four workers.

0.25

Tremulant: Latency in Serverless Platforms beyond the 99th Percentile

cs.DC ·2026-10-13 ·Linus Gustafsson, Trever Schirmer, David Bermbach
similarity 0.66
Abstract

Function-as-a-Service (FaaS) increases developer productivity by removing operational concerns such as managing hardware and software runtimes. Because function instances are started on demand, some requests must wait for a new instance, called a cold start. These are a common obstacle to more widespread adoption of FaaS, as they lead to unpredictable latency spikes that are not acceptable for some use cases. To solve this, many cold start optimizations have been implemented. However, comparatively little attention has been directed towards the slowest warm starts. With improvements in cold start times, slow warm starts emerge as a new problem for extreme tail events. By analyzing a dataset of several million requests across seven FaaS platforms, we show that the long tail of slow warm starts can be hundreds of times slower than their median performance, while a virtual machine we use as a baseline is substantially more consistent. We confirm this across four independent public datasets spanning 2020 to 2025, so the effect is neither new nor specific to one platform. Moreover, the warm share among the slowest calls starts to rise again at the highest percentiles: the most extreme latencies are not only cold starts, and cold-start mitigation alone cannot remove them.

0.18

Performance Study of Serverless Workloads in Confidential Virtual Machines

cs.DC ·2026-09-03 ·Rikesh Niroula, Jianchen Shan, Xiaoning Ding
similarity 0.63
Abstract

Confidential serverless computing is rapidly emerg- ing as a critical paradigm for application domains requiring strong confidentiality guarantees, such as healthcare, finance, and machine learning. To enable this paradigm in untrusted cloud environments, Confidential Virtual Machines (CVMs) provide isolation by encrypting the entire guest memory, and thus securing serverless workloads against host-level access and inter- ference. However, the implications of CVMs for serverless systems remain insufficiently understood. This paper presents an empirical study of serverless work- loads in CVMs, systematically covering memory efficiency and runtime overhead on both warm-starts and cold-starts. Our results show that CVMs incur substantial memory overhead because encrypted memory disables cross-VM page deduplication and reduces memory reclaimability, thereby limiting warm- container capacity under a fixed memory budget. Runtime overhead in warm-starts is workload dependent. Workloads with frequent VMEXITs, particularly idle transitions, suffer substantial slowdowns. These slowdowns are further amplified by common serverless deployment practice that couples vCPU allocation to memory size, as higher-memory configurations expose more vCPUs than some functions can use effectively. For cold starts, the study focuses on container creation, a common and major contributor to startup latency. Motivated by the memory-efficiency results, we consider the deployments in which multiple containers of the same function are consolidated within the same CVM to improve efficiency, and show that selectively relaxing certain isolation mechanisms in this setting can substan- tially reduce startup overhead. These results clarify the main performance tradeoffs of confidential serverless computing and suggest practical ways to improve efficiency and latency.

0.05

Benchmarking Linux CGroup and Schedule Idle

cs.OS ·2026-11-18 ·Anna Arpaci-Dusseau
similarity 0.63
Abstract · 11 pages, 10 figures

cgroups are a Linux kernel primitive that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O) of a collection of processes. Many provider-side serverless systems (Mitosis, gVisor, Havest VM, FAASM) leverage linux containers based around cgroups/namespaces for their performance isolation. However, how well does this primitive map to the needs of serverless platforms? How does the linux CFS scheduler deal with cgroups and SCHED IDLE and what are the core sources of high tail latency? Would a new primitive design fit the needs of serverless isolation better? In this work we take the first step to answering these questions by profiling the current behavior of cgroups and a sched idle policy via a thorough benchmarking study. We emulate the setting of Performance-Critical (LC) server in the presence of an intensive background (BE) process and note the impact of each primitive on the performance of the server. We show it is non-trivial to balance the potential tail-latency impact on a LC server with the throughput of BE tasks. Furthermore, there are many variables to adjust for the developer between hard maximum bandwidth limits, weighted cpu shares, the presence of the sched idle policy, or additionally "niceness" priority in the scheduling policy.

0.26

Reducing Cold-Start Latency in Serverless Applications via Dynamic Slicing

cs.DC cs.PLcs.SE ·2026-09-12 ·Georgios Alexopoulos, Konstantinos Karakatsanis, Nikolaos Alexopoulos, Dimitris Mitropoulos, Thodoris Sotiropoulos
similarity 0.61
Abstract

We present PyXtrim, a system that reduces the cold-start latency of serverless applications through debloating. We focus on Python, a dominant language for serverless applications whose dynamic features and extensive use of native extensions make traditional static debloating particularly challenging. PyXtrim frames debloating as a dynamic slicing problem, using the application's externally visible behavior as the slicing criterion. Everything outside the resulting slice is removed, both from the application and its dependencies. Our key technical contribution is that the slice is computed by a cross-language dynamic dependence engine that tracks data and control dependences across Python and native code and identifies operations that interact with the operating system, which form the slicing criterion. As a result, PyXtrim can effectively handle real-world applications that rely on dynamic features such as reflection, interoperate with native code and interact with system resources. Across 31 applications on AWS Lambda, PyXtrim reduces cold-start latency by 21.7% and peak memory usage by 17.1% at the median. This is more than double the reduction achieved by the state of the art, while debloating each application in minutes.

0.17

LLM Fine-Tuning Workloads: Characterization, Challenges, and Opportunities for Serverless AI

cs.DC ·2026-10-29 ·Radu Nicolae, Daniele Lotito, Alexandru Iosup
published similarity 0.61
Abstract · Technical report; extended version of the paper published at the Workshop on AI and Serverless (WoAIS '26), co-located with ACM/IFIP Middlew…

Serverless execution of Large Language Models (LLMs) is becoming important, reflecting their wide societal use. An increasingly present stage in LLM pipelines is fine-tuning, which specializes trained models and improves inference performance. Serverless operation of LLM fine-tuning workloads shifts operational concerns from the user to the provider. For these new workloads, providers face important operational complexities, such as performance issues, resource waste, and even service failure due to misconfiguration. However, optimizing such workloads is still at an early stage, and, relevant to this work, relatively little is currently known about how LLM fine-tuning workloads run in practice. In this work, we address the knowledge gap by characterizing the execution of LLM fine-tuning workloads on clusters equipped with different GPUs, running LLM models at various scales. We analyze IBM's LLMFineTuningBench, a large-scale, diverse profiling dataset of 30,920 fine-tuning experiments that records both successful and failing configurations. We provide quantitative insight into throughput (expressed in tokens/s), scaling, failure, and potential for performance optimization for four commonly used GPU types, in setups of 1 to 32 GPUs. We find that (1) job feasibility (run or fail) depends on workload properties, with failure ratios rising from 52% for small batch sizes (16-64) up to 95% for the largest (1,024); (2) throughput grows logarithmically with batch size; (3) scaling is near-linear when the workload is deployed intra-node, and optimizations such as RoCE substantially improves efficiency for inter-node deployment (e.g., 0.72 vs. 0.26 efficiency without RoCE); and (4) system-level optimizations can outweigh application-level ones (RoCE, up to 2.9$\times$, vs. Fast MoE, 1.7$\times$).

0.27

Energy-Proportional Serverless Computing on SBC Clusters

cs.DC ·2026-11-10 ·Yuliya Sasinskaya, Natalie Carl, Niklas Kowallik, David Bermbach
similarity 0.60
Abstract · Accepted for publication at the 4th International Workshop on Middleware for the Computing Continuum (Mid4CC '26)

Serverless computing promises to run code only on demand, yet the servers beneath it draw most of their power while idle, so energy consumption does not scale to zero with the workload. We show that the statelessness of serverless functions makes it possible to extend scale-to-zero from function instances to physical nodes, enabling energy proportionality. Our platform, built on tinyFaaS for clusters of single-board computers, treats node power state as part of worker selection, powering workers off when idle and back on under load. On a physical cluster, our approach reduces mean power draw by 39.6 % on a bursty, trace-based workload and by 46.7 % on a sparse one, without a detectable change in median latency on the bursty workload. The cost lands in the tail, where invocations that arrive while all workers are off wait about a minute for a node to boot. Under continuous load, the platform behaves like an always-on cluster.

0.35

Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving

cs.DC cs.AIcs.LG ·2026-11-10 ·Wen Huang, Jack Liu
similarity 0.60
Abstract · 12 pages, 8 figures

LLM agents drive inference engines with closed-loop sessions whose context grows every turn, and serving them economically depends on prefix caching: a hit costs an incremental prefill of about a thousand tokens, a miss recomputes a 10--40K-token context. We show that this makes prefix-cached agent serving metastable. With an identical base workload, a few minutes of extra arrivals leaves a vLLM server in a degraded regime--a ~10% hit ratio, about half the throughput, and a backlog that grows for the remaining 70 minutes--while its no-burst twin stays healthy (9 of 12 pairs; none in the reverse direction). The regime appears at loads the cache-warm server sustains, is absent without prefix caching, and also appears with Qwen3-32B on SWE-bench agent trace shapes and on SGLang, which tolerates larger triggers. Slower service keeps more sessions in the system, their working set outgrows the KV pool, and every turn becomes a recomputation; a first-order workload-balance condition, calibrated independently, brackets when the regime can drain. Getting out is the harder problem: throttling new admissions recovered 1 of 7 matched collapsed runs, while suspending admitted sessions against a capacity-based limit recovered all of them within minutes. A CPU offload tier delays the collapse but does not prevent it. A budgeter that bounds the resident KV demand of active sessions to the pool, with thresholds that need no per-workload tuning, prevented the failure in every run we tested.

0.15