CS Paper Radar

Beyond the Limits: Flexible and Congestion-Aware Cluster Scheduling for the Cloud

Oliver Larsson, Thijs Metsch, Cristian Klein, Erik Elmroth

cs.DC ·submitted 2026-08-10 08:53 UTC · v1

Abstract

Workload scheduling in cloud environments often relies on simplistic assumptions about application resource needs and hardware utilization. Overlooking application-level performance objectives and hardware resource contention that leads to inefficient resource usage and degraded performance. This paper addresses two key limitations of current approaches. First, unnecessarily strict enforcement of service level objectives (SLOs) often leads to resource underutilization and poor energy efficiency. Second, lack of congestion awareness in shared resources such as last-level cache (LLC) and memory bandwidth. In this paper, we propose two complementary strategies to address these limitations: (i) integrating soft SLO limits that allow controlled overcommitment and tolerate minor, transient violations to improve cluster efficiency, and (ii) introducing resource-aware scheduling and rescheduling based on real-time congestion insights for shared resources such as last-level cache (LLC) and memory bandwidth. Our results show that soft SLO limits reduce corrective rescheduling actions by 49% compared to hard-limit approaches while maintaining acceptable performance guarantees. Additionally, resource-aware scheduling decreases node-level congestion by 8% and further mitigates SLO violations, demonstrating the effectiveness of incorporating application-level flexibility and hardware-level insights into scheduling and rescheduling decisions.

Comment
10 pages, 9 figures, 3 tables

Closest papers

Nearest neighbours by meaning across the whole index.

Holistic Buoyancy-Driven Cloud Native Resource Management

cs.DC ·2026-10-19 ·Oliver Larsson, Thijs Metsch, Cristian Klein, Erik Elmroth
similarity 0.79
Abstract · 12 pages, 8 figures, 3 tables

Efficient resource management is a challenge in multi-tenant cloud environments as workloads compete for shared resources such as last-level cache and memory bandwidth, whose congestion can change dynamically as workload composition and resource demands evolve. Existing resource management approaches often rely on static resource requirements or prior workload profiling, limiting their ability to respond to changing resource contention as required in the cloud. In this paper, we leverage workload buoyancy, a runtime framework for capturing workload sensitivity to individual resources and application-level performance, to enable adaptive scheduling and resource management. We propose a holistic approach that incorporates buoyancy into workload scheduling, node-local resource allocation, and workload rescheduling throughout the workload lifecycle. We implement our approach as a Kubernetes-native prototype and evaluate it on a multi-node cluster using diverse cloud workloads under varying levels of resource contention. In a high-congestion scenario, our approach increases the percentage of workloads satisfying their SLOs from 7.74% to 87.8% on average compared with the Kubernetes baseline, while reducing resource congestion, the severity of SLO violations, and cluster fragmentation using limited rescheduling.

0.23

Performance and Cost-Aware Cache Provisioning

cs.PF ·2026-08-10 ·Ridwanul Tanvir, George Kesidis
similarity 0.71
Abstract

While traditional cache policy evaluations fix capacity - often at 0.1% of the dataset - and measure the resulting hit rate, practical edge-cloud deployments require balancing both storage and computational overhead as billed resources. Consequently, system operators frequently focus on a different objective: determining the minimum cache size needed to satisfy a specific Service-Level Objective (SLO) hit-rate. This paper explores this SLO-centric paradigm by analyzing the minimum capacity and execution time each policy requires to hit a defined target. Additionally, we show that dynamically adjusting the segment ratio in segmented policies based on historical workload patterns enhances efficiency. Through evaluations across real-world and synthetic traces, we present a novel hybrid segmented policy that reduces capacity requirements while keeping processing costs low.

0.04

PANEM: A Heuristic Latency Model

cs.AR cs.PF ·2026-08-28 ·Lucas Crowthers, Mahesh Madhav
published similarity 0.68
Abstract · 12 pages, 23 figures, Presented at the 2026 IEEE International Symposium on Workload Characterization (IISWC 2026), Boulder, Colorado

Accurate pre-silicon memory modeling is essential for achieving meaningful representation of workloads on cloud-class many-core processors. Existing options force a poor tradeoff between fidelity and speed: fixed-latency models are fast but misleading, while cycle-accurate DRAM models are costly and difficult to scale across large study spaces or onto single-core environments. This paper presents PANEM, a lightweight event-driven heuristic model that has guided four generations of commercial server-core development at Ampere Computing. PANEM converts bandwidth-latency characterization data into a dynamic request-bytes/latency response, allowing miss latency to adapt to transient demand, queuing pressure, and read/write mix during simulation. Integrated into a single-core flow with configurable system-loading assumptions, PANEM enables realistic bandwidth constraints and contention-aware latency behavior without sacrificing throughput. Across a broad cloud workload trace suite, PANEM avoids the optimistic and pessimistic biases of fixed-latency baselines, yields more reliable conclusions for prefetching and dynamic throttling studies, and materially improves core-resource sizing decisions. These results show that a calibrated, contention-aware abstraction can deliver practical predictive value for industrial design-space exploration at simulation costs similar to fixed-latency models.

0.46

Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving

cs.DC cs.LG ·2026-09-16 ·Dev Bali, Soujanya Ponnapalli, Yichuan Wang, Natacha Crooks, Scott Shenker +1 more
similarity 0.68
Abstract

LLM serving is typically offered as a shared, multi-tenant service, where high-demand workloads from one client can cause latency SLO violations for others. Existing solutions for performance isolation equalize client throughput in the long run, for example through queueing and batching fairness. However, these approaches do not provide latency isolation guarantees; as a result, well-behaved clients can still experience significant degradation to their token-level latencies. In this paper, we present FairInference, which provides the novel δ-token fairness guarantee: for a well-behaved client, if a token is generated in d time units in isolation, it will be generated within d + δ time units in multi-tenant execution, providing strong latency isolation guarantees for LLM serving. To achieve this, FairInference addresses a key challenge of LLM serving: bounding delays from sharing GPU resources without support for fine-grained scheduling or resource allocation. In FairInference, the scheduler enforces per-token deadlines, while bounding the delays from GPU compute sharing and accounting for the additional delays introduced by the shared KV caching in GPU memory. We show that FairInference effectively bounds token-level latency spikes for well-behaved clients and improves overall throughput compared to state-of-the-art LLM serving systems.

0.46

PackServe: SLO-Aware Request Scheduling for Agentic LLM Serving at Scale

cs.DC ·2026-09-27 ·Zhiyuan Tan, Dejiang Zhu, Jingzhe Jiang, Yihao Zheng, Yang Tian +2 more
similarity 0.68
Abstract · 13 pages, 13 figures

Request scheduling is a key challenge in large-scale clusters serving agentic large language model (LLM) workloads. An effective scheduler must preserve key-value cache (KVC) reuse across long, shared prefixes, meet token-level latency service-level objectives (SLOs), and minimize GPU resource footprint. Existing schedulers struggle to reconcile these requirements: request consolidation can sacrifice cache locality and increase prefill/decode interference, compromising both SLO attainment and resource efficiency. We present PackServe, a scheduler designed to reduce resource costs while meeting latency SLOs for agentic LLM serving. PackServe uses compact white-box models to predict latency under prefill/decode interference. Guided by these predictions, it packs requests onto fewer serving instances while preserving KVC reuse and SLO constraints, trading available latency headroom for improved per-instance throughput. Evaluation on 64 NVIDIA H20 GPUs shows that PackServe uses up to 16.8% and 24.6% fewer GPU-hours than state-of-the-art schedulers under 30-ms and 50-ms TPOT targets, respectively, while meeting the target TPOT objectives. PackServe has also been deployed in our production cluster comprising over 1000 GPUs, where it reduces the resource footprint by 34.7% compared with the original production scheduler.

0.21

Carbon-aware Resource Management for Latency-Sensitive Cloud Computing Environments: A Taxonomy and Future Directions

cs.DC ·2026-09-03 ·Tharindu B. Hewage, Shashikant Ilager, Maria Rodriguez Read, Rajkumar Buyya
survey similarity 0.67
Abstract

The proliferation of cloud-based latency-sensitive workloads, such as autonomous driving, real-time healthcare services, industrial automation, immersive interactive services, and intelligent AI-based applications requires infrastructures tuned to meet stringent latency constraints. Today, these workloads are reshaping the cloud from a general-purpose computing platform into a diverse ecosystem of workload-specific cloud environments. As demand for latency-sensitive workloads grows, cloud providers must scale their infrastructures accordingly, adversely increasing the carbon footprint and challenging net-zero goals. However, the strict performance requirements and rigid deployment patterns used to satisfy latency Service Level Objectives (SLOs) constrains conventional carbon optimization. Therefore, novel carbon-aware techniques are needed to exploit application-specific opportunities for reducing both operational and embodied carbon footprints while preserving latency guarantees. This survey examines carbon-aware resource management for latency-sensitive cloud computing environments. We introduce a holistic taxonomy that organizes existing techniques by carbon management objective, workload management mechanism, control architecture, system tier, latency requirements, and deployment stage. Then, we use this taxonomy to classify and critically analyze existing approaches, revealing the interactions and trade-offs between latency guarantees and operational and embodied carbon reduction. Finally, we identify limitations in the literature and present a structured research agenda for the design of sustainable latency-sensitive cloud systems.

0.09

From Overloaded to Guaranteed: High-Throughput Multi-SLO Enforcement for LoRA-Assisted On-Premise LLM Deployment

cs.CL cs.DC ·2026-10-04 ·Zeshen Zhang, Han Zhao, Weihao Cui, Quan Chen, Yu Liu +7 more
similarity 0.67
Abstract · 22 pages, 16 figures, 4 tables

As Large Language Models (LLMs) become essential in privacy-sensitive sectors like hospitals and government agencies, the on-premise LLM servers offer a cost-effective and secure alternative to public cloud services. However, these resource-constrained servers struggle to guarantee heterogeneous Service Level Objectives (SLOs) when serving multiple LoRA-adapted services simultaneously. Existing serving frameworks suffer from severe SLO violations due to the computational overhead of LoRA layers and the rigid nature of batch scheduling. To address this, we propose HALO, a scheduling method tailored for LoRA-assisted on-premise LLM deployment. HALO introduces two key innovations: a spatial multiplexing strategy that overlaps Base and LoRA computations by partitioning GPU Streaming Multiprocessors (SMs), and an SLO-aware scheduler that decouples request execution based on "request-level slack." By prioritizing urgent tasks and utilizing idle budget for traffic shaping, HALO significantly mitigates resource contention. Our evaluation demonstrates that HALO minimizes SLO violations while improving throughput compared to state-of-the-art baselines.

0.14

Task Offloading and Resource Allocation for Deterministic Service Provisioning in the IoT-Edge-Cloud Continuum

cs.NI ·2026-10-17 ·K. Aghababaiyan, B. Coll-Perales, J. Gozalvez
similarity 0.67
Abstract

Future networks will seamlessly integrate computing and communication resources within a unified ecosystem of IoT devices, edge nodes, and cloud servers, forming an IoT-edge-cloud continuum. Within such an ecosystem, tasks can be dynamically offloaded to leverage resources across the continuum and efficiently scale with demand. Task offloading has traditionally focused on minimizing task execution latency or energy consumption. In contrast, we propose a deterministic task offloading and resource allocation scheme for the joint management of communication and computing resources to support services with deterministic service-level requirements. The proposed scheme aims to maximize the number of tasks completed on time, rather than minimizing the latency of individual tasks. To do so, it exploits variable task latency requirements to distribute the workload across the continuum. It can also prioritize high-priority tasks under resource scarcity. Numerical evaluations demonstrate that a deterministic task offloading strategy scales more effectively and can better guarantee timely task completion by balancing resource utilization across the continuum.

0.22