CS Paper Radar

Slant: Shaping Traffic for Resource-Efficient Microservices

Congwei Zhang, Chieh-Jan Mike Liang, Mengbai Xiao, Yifei Zou, Shulun Wu, Xiangyu Gao, Xiuzhen Cheng, Dongxiao Yu

cs.DC ·submitted 2026-10-28 19:52 UTC · v1

Abstract

Microservice architectures rely on load balancers to distribute requests across service replicas, yet existing designs largely optimize for balanced routing without considering how routing shapes autoscaling signals and resource efficiency. We present Slant, a resource-aware load balancing framework that intentionally creates safe traffic imbalance to improve resource utilization. Slant profiles service-specific critical points, computes conservative multi-dimensional capacity, derives imbalance-aware routing weights that maintain hot and cold pods, and falls back to safer behavior under anomalies. By exposing clearer utilization signals, Slant works with existing vertical pod autoscalers (VPAs) without modification. Experiments on three microservice benchmarks show that Slant reduces allocated CPU cores by up to 26\% while lowering P99 latency by up to 18\% compared to Round-Robin. Under dynamic workloads, Slant triggers VPA scale-up 45\% faster and completes scale-in 40\% faster. Under anomalies, it recovers 40\% faster from pod failures and limits latency amplification to 1.8$\times$ versus 2.6$\times$ during downstream slowdowns.

Closest papers

Nearest neighbours by meaning across the whole index.

SLO-Scaler: Uncertainty-Aware SLO-Driven Autoscaling for Microservices

cs.DC ·2026-08-18 ·Shuo Wang, Xiaoxuan Sun, Shao-yu Huang, Bencheng Su, Shuo Xu +1 more
benchmark similarity 0.67
Abstract

Autoscaling microservice-based applications to satisfy Service Level Objectives (SLOs) remains challenging due to bursty workloads, cascading latency across service dependencies, and cold-start overhead. Existing approaches such as the Kubernetes Horizontal Pod Autoscaler (HPA) rely on threshold-based CPU or memory metrics, which react too slowly to traffic spikes. Recent predictive methods improve responsiveness but generate point forecasts that ignore prediction uncertainty, leading to over-provisioning or oscillatory scaling. We propose SLO-Scaler, an uncertainty-aware autoscaling framework that predicts short-horizon request rates, tail latency, and SLO violation probability using a Bayesian LSTM model. SLO-Scaler integrates confidence-interval-based scaling decisions with a dependency graph analysis module that localizes bottleneck services, avoiding unnecessary whole-chain scaling. We evaluate SLO-Scaler on the DeathStarBench Social Network benchmark deployed on Kubernetes under periodic, bursty, and long-tail traffic patterns. Under bursty traffic, SLO-Scaler reduces the SLO violation rate by 29-56%, lowers the average replica count by 18-33%, and decreases scaling event frequency by 38-59% compared with the baselines, while achieving lower tail latency.

0.33

InferScale: Kubernetes-Native Predictive Autoscaling for SLO-Aware AI Inference

cs.DC ·2026-10-25 ·Wen Huang
published similarity 0.66
Abstract · 8 pages. Published in IEEE CLOUD 2026

Kubernetes Horizontal Pod Autoscaler (HPA) is widely used for inference services, but reactive scaling is often too slow when pods take tens of seconds to become ready. The problem gets worse in shared clusters, where independent HPAs compete for the same resource budget and can hurt different tenants from run to run. We present InferScale, a Kubernetes-native autoscaler that combines inference-aware signals, short-horizon demand prediction, SLO-state control, fast-path spike handling, and tenant-priority headroom without replacing the default scheduler. We evaluate InferScale on Azure Kubernetes Service in two settings. On a real ResNet-50 ONNX workload with 21-second cold start, repeated ramp-burst runs improve SLO compliance from 47.8% +/- 1.5% under a stronger external-metric HPA to 65.3% +/- 9.0% under InferScale. In repeated controlled multi-tenant contention experiments, InferScale cuts Gold-tier SLO standard deviation from 16.1 to 1.2 percentage points and makes contention outcomes follow the priority policy more closely. Repeated single-service ablations show 6-8 percentage point degradation, with an exploratory priority ablation showing a 27-point drop under contention. The main takeaway is that, in startup-sensitive inference serving, the practical value of prediction lies less in precise forecasting than in moving the controller early enough to offset cold-start delay.

0.25

Offering Microsecond-Scale Cross-VM Core Elasticity on Colocated Lightweight Virtual Machines

cs.DC cs.OS ·2026-08-12 ·Yibo Yan, Seo Jin Park
similarity 0.63
Abstract · Under review

Serverless platforms commonly colocate many diverse workloads, each in a fast-booting, memory-lean virtual machine (VM), to improve deployment density. Overprovisioning each VM for its peak protects tail latency during traffic bursts but hurts density; maintaining high density while effectively protecting tail latency requires the infrastructure to be able to shift physical cores, at a microsecond timescale, to whichever latency-sensitive VM is bursting and reclaim them as the burst subsides. No VM substrate delivers this: conventional VMs resize a guest's cores only through a millisecond-scale vCPU hot-plug path, Firecracker fixes a VM's core count at boot, and the ultralight VMs that boot fastest drop multicore execution entirely. We present HyperFlux, a commodity-KVM ultralight VM substrate that makes a VM's parallelism width (the number of physical cores backing it) elastic at runtime. We show that HyperFlux can move a core across VMs in merely 13$μ$s, even when forcibly reclaiming it from a busy donor, orders of magnitude faster than vCPU hot-plug. A HyperFlux VM incurs only a 3.2MB memory footprint and can cold-boot in 1.37ms, on par with the fastest-booting ultralight VMs, while uniquely supporting multicore parallelism. Under colocation, it can reduce high-priority VMs' tail latency by up to 10x under high load compared to static core-sharing with Firecracker and Cloud Hypervisor, and deliver a lower and more stable tail latency compared to using cgroup and vCPU hot-plug under changing load bursts.

0.38

FluidPD: In-Place Elasticity for SLO-Aware Prefill-Decode Disaggregated LLM Serving

cs.AI ·2026-10-02 ·Kartik Ramesh, Kaidi Fu, Zihan Zheng, Jiahuan Yu, Fabio Oliveira +2 more
similarity 0.63
Abstract · 13 pages, 11 figures

Prefill-decode disaggregation is becoming a common architecture for LLM serving because it separates two phases with distinct execution patterns and SLO objectives. Existing systems typically combine a fixed prefill/decode worker ratio with request routing across workers. However, real-world workloads exhibit both short bursts and sustained shifts in the prefill-to-decode demand ratio. As a result, a configuration that is well provisioned at one time may quickly become mismatched, causing latency SLO violations even when idle capacity exists elsewhere. Existing autoscaling mechanisms can add capacity, but they react slowly, require spare GPUs, and do not directly address short-timescale phase imbalance. We present FluidPD, a P/D-disaggregated serving system that provides SLO-aware in-place elasticity. FluidPD introduces two complementary mechanisms. FluidToken handles transient imbalance by offloading a bounded portion of prefill computation to decode workers when decode-side slack is available. FluidRole handles sustained imbalance by reassigning running workers between prefill and decode roles in place, avoiding model reload and engine restart. Both mechanisms are guided by lightweight pressure indices that expose prefill and decode-side resource pressure before they appear as SLO violations. Across production Azure trace workloads, FluidPD improves overall SLO attainment over static SGLang by up to 94.6 percentage points, demonstrating that SLO-aware in-place P/D elasticity improves service quality without provisioning additional workers.

0.16

Cross-Model Autoscaling for Shared LLM Serving

cs.DC ·2026-09-24 ·Xin Zhang, Xianyan Xie, Zhen He, Xijin Yin, Xingtong Lin +4 more
code/data similarity 0.63
Abstract · 11 pages, 6 figures

Multi-model LLM serving is moving toward shared MaaS clusters, where co-hosted models compete for a fixed GPU budget while each model experiences time-varying demand and must satisfy its own latency SLO. Existing LLM autoscalers remain largely model-local: their signals expose local runtime activity or delayed latency outcomes, and their scale-up or provisioning decisions do not directly determine how shared capacity should be allocated across competing models. We present the Token-service-share Rebalancing Engine (TRE), a control-plane framework for hot-switched multi-model LLM serving. TRE introduces Token Service Share (TSS), a calibrated, demand-normalized signal that estimates effective token service per active or queued request and yields a comparable health score across heterogeneous models and SLO classes. Guided by TSS, TRE coordinates bounded receiver--donor capacity movement under a fixed GPU budget: it separates fast rescue from slower rebalancing and incrementally reallocates active replicas toward models with the largest calibrated service deficits. We implement TRE on a Kubernetes-based hot-switch serving stack without modifying the inference scheduler. Across seven LLM serving traces, TRE reduces P95 end-to-end latency by 11.9--79.0\% and P99 latency by 12.5--72.6\% compared with a state-of-the-art KV-cache-based reactive autoscaler running on the same hot-switch runtime. The gains hold on both targeted stress probes and production-derived conversation/code traces, where TRE reduces P95/P99 latency by 50.8/63.7\% and 79.0/72.6\%, respectively. These results show that effective hot-switched autoscaling requires not only fast replica actuation, but also calibrated service-deficit signals and coordinated cross-model capacity arbitration. Our code and artifacts are available at https://github.com/zxzx9898/Token-service-share_Rebalancing_Engine.

0.51

A Kubernetes-Native Request Router for Quality-Aware Inference Serving in the Computing Continuum

cs.DC ·2026-09-17 ·Ignjat Karanovic, Pantelis A. Frangoudis, Ivan Čilić, Ivana Podnar Žarko, Schahram Dustdar
similarity 0.62
Abstract

We introduce Adaptive Score-based Routing Balancer (ASRB), a dynamic, score-based request routing mechanism for Kubernetes-based service deployments over the computing continuum. ASRB jointly considers infrastructure-level information, response time measurements, and application-level quality indicators, with a particular focus on serving Machine Learning (ML) workloads. For these workloads, ASRB balances requests over service instances deployed in the continuum, following service provider-defined policies encoded as weighted combinations of QoS criteria to flexibly address latency-accuracy trade-offs. To drive routing decisions and swiftly adapt to changes in the operating environment, ASRB monitors a range of runtime metrics across multiple system layers. To deal with the associated monitoring overhead, particularly important for large-scale deployments, it selectively and adaptively controls monitoring intensity without sacrificing on routing quality. ASRB is implemented without requiring any modifications to Kubernetes, making it straightforward to deploy and operate in existing cluster environments. Our testbed experiments demonstrate the versatility of ASRB: When tuned for latency reduction, it achieves at least 10 ms lower mean response time compared with latency-oriented state-of-the-art routing mechanisms, while it achieves higher accuracy when this is prioritized through specific configurations, thus enabling flexible and operator-controllable trade-offs. At the same time, it attains reduced failure rates, higher responsiveness to changes in the operating environment, and up to ~70% less monitoring cost than relevant state-of-the-art solutions, at the potential expense of only a modest latency penalty in some configurations.

0.28

Holistic Buoyancy-Driven Cloud Native Resource Management

cs.DC ·2026-10-19 ·Oliver Larsson, Thijs Metsch, Cristian Klein, Erik Elmroth
similarity 0.61
Abstract · 12 pages, 8 figures, 3 tables

Efficient resource management is a challenge in multi-tenant cloud environments as workloads compete for shared resources such as last-level cache and memory bandwidth, whose congestion can change dynamically as workload composition and resource demands evolve. Existing resource management approaches often rely on static resource requirements or prior workload profiling, limiting their ability to respond to changing resource contention as required in the cloud. In this paper, we leverage workload buoyancy, a runtime framework for capturing workload sensitivity to individual resources and application-level performance, to enable adaptive scheduling and resource management. We propose a holistic approach that incorporates buoyancy into workload scheduling, node-local resource allocation, and workload rescheduling throughout the workload lifecycle. We implement our approach as a Kubernetes-native prototype and evaluate it on a multi-node cluster using diverse cloud workloads under varying levels of resource contention. In a high-congestion scenario, our approach increases the percentage of workloads satisfying their SLOs from 7.74% to 87.8% on average compared with the Kubernetes baseline, while reducing resource congestion, the severity of SLO violations, and cluster fragmentation using limited rescheduling.

0.23

A Technique for Load Shifting Low-latency Applications in Multi-Region Renewables Harvesting via SMT Core Pooling

cs.DC ·2026-09-03 ·Tharindu B. Hewage, Shashikant Ilager, Maria A. Rodriguez, Rajkumar Buyya
similarity 0.61
Abstract

Load shifting across geographic regions to chase intermittent renewable energy availability is commonly used in reducing cloud infrastructure carbon footprint. However, it often omits low-latency applications due to high latency variances of wide area networks (WAN) that interconnect regions. This paper addresses accommodating low-latency applications into load shifting by minimizing their shifting across the WAN. We propose a technique using a hardware-software co-design approach. At the hardware level, we conduct server load matching over renewables supply peaks and valleys by deep idling physical cores in two otherwise identical server pools, with one enabling simultaneous multi-threading (SMT) in CPUs. In return, we achieve a static set of logical cores amidst energy supply dynamics, reducing the probability of workload shifting. At the software level, we efficiently chase the static set of cores for low-latency applications within regions while prioritizing best-effort applications to accommodate shifting requirements across WANs. Our approach exploits the lower performance compromise of SMT cores due to their hardware multi-threading. We implement the proposed technique with OpenStack and CPU idle states and evaluate its performance on a real experimental testbed with Azure VM arrival traces. Results show an 80% reduction in offloading low-latency VMs and a 43.81% reduction in coefficient of variation of p90 end-user latency while having a worst-case latency compromise of 11.97% due to SMT cores.

0.12