CS Paper Radar

Bounding Queue Waits for Fair Scheduling of Bursty HPC Workloads via Hierarchical Algorithms

Karaen Senthilkumar, Suman Raj, Maxime Gonthier, Kyle Chard, Donpaul Stephens, Ian Foster

cs.DC ·submitted 2026-10-23 01:06 UTC · v1
hot direction

Abstract

HPC clusters are shared among many projects, each holding an allocation of node-hours, and ensuring fairness is becoming harder. Foundation models workflows and automated pipelines have made job submissions bursty and concentrated among a few heavy users, so occasional users absorb queueing delays created by frequent ones, with no indication of how long their jobs will wait. Existing schedulers either forget past usage, protect only the job at the head of the queue, or assume that running jobs can be preempted. We propose HyDRA, a Hybrid Deadline and Ranking-Aware scheduler that enforces fairness among users while bounding the queue wait of every job. HyDRA assigns each job a deadline on arrival based on its size, the priorities of its project and user, the current backlog, and how far its project and user are from their entitled share. It ranks waiting jobs by a priority score that grows with waiting time, serves jobs close to their deadlines first, and backfills a job only when no deadline, or fairness limit is violated. We evaluate HyDRA on workloads derived from a year of ALCF Polaris job traces, against conventional schedulers and variants inspired by PCS, Themis, and the PBS job-sort formula used at ALCF. HyDRA achieves $2\times$ the project-level Jain's fairness of the best baseline, starts 47\% of jobs by their deadlines compared with at most 36\% for the baselines even for largest workloads, and lowers the 95th-\%ile bounded slowdown by 4.2$\times$ relative to the next best baseline, at a moderate cost in utilization.

Comment
12 pages, 9 figures, 4 tables

Closest papers

Nearest neighbours by meaning across the whole index.

Dispatch Control for Priority-Based, Multi-Tenant Workflows under Multi-Resource External Rate Limits

cs.DC ·2026-11-02 ·Stanislav Kosorin, Trever Schirmer, Leonard Wossnig, David Bermbach
similarity 0.71
Abstract · Author version of paper accepted in 15th International Workshop on Cloud and Edge Computing, and Applications Management (CloudAM '26)

Workflow engines orchestrate pipelines of activities, which often call external APIs, on a fixed pool of worker slots, the threads that run them. When an API enforces a hard rate limit shared by all tenants and priorities, a throttled call that retries after a backoff keeps its worker slot and starves other tenants. Large Language Model provider APIs are particularly complex: their per-organization caps are low and bind independently on requests and tokens per minute. The operator can neither reconfigure the cap nor observe saturation before a request fails, so classical fair-queuing assumptions do not hold. We present Backpressure, a worker-side admission gate. It checks each call's cost atomically against shared admission state and enforces fairness over requests and tokens. A denied call releases its worker slot and waits on a durable timer, so waiting no longer blocks execution capacity. We compare against a naive retry, the AI gateway LiteLLM, and the built-in throttle of the workflow engine Temporal. On a workload with three priority classes, Backpressure raises fairness (Jain's index over each queue's fraction of its entitled share) from 0.60 to 0.89, where 1.0 is ideal. It also cuts high-priority tail latency roughly 8x. When tokens are the binding limit, all request-based baselines serve every priority equally, whereas Backpressure keeps the priority order and reaches a dominant-resource fairness of ~0.99. Admission costs one round trip to shared state, adds no measurable latency when the cap does not bind, and fails open if shared state is unavailable.

0.29

CheckCo: Checkpoint-Coordinated Load Balancingfor Skewed Event Streams at Scale

cs.DC ·2026-11-17 ·Muhammad Anis Uddin Nasir, Reza Sadraei
similarity 0.67
Abstract · This paper is submitted to ICDE 2027 Industry and Application Track

Event streams are rarely uniform: a few dominant keys can generate orders of magnitude more events than the rest, overloading the workers that receive them and capping throughput. We present CheckCo, a checkpoint-coordinated load-balancing algorithm built on a dynamic oracle that estimates per-key frequencies from the live stream, with no prior statistics. Workers count events per key locally; at each checkpoint barrier, the counts are merged into a partitioning function that divides each key into a number of splits roughly proportional to its estimated frequency and packs the splits onto workers, and this function routes the next checkpoint window. We derive approximation bounds on the resulting load imbalance from classical scheduling results and bound its total key fan-out. On a production pipeline processing over 100 billion events per day, the most loaded worker receives only 9.8% more events than the average over a day (imbalance ratio 1.10), against 8.03 for key-based routing in simulation; in simulation, CheckCo also needs less than half the fan-out of earlier partitioners that reach comparable balance. Shifts in the key distribution are reflected in routing from the next checkpoint on, and the load stays balanced through job restarts.

0.20

Vesta: Queueing-Aware LLM Scheduling for Real-World Workloads

cs.DC ·2026-10-10 ·Lancheng Zou, Bowen Li, Wei Mao, Wenbin Ye, Lecheng Li +4 more
similarity 0.65
Abstract

Agentic applications generate multi-turn sequences of LLM requests, with each new turn extending the context accumulated from prior interactions and tool calls. Prefix caching reuses the KV cache of earlier turns and avoids redundant prefill, but routing every turn to its cache-affine instance can create severe local queues while capacity remains idle elsewhere. Remote KV cache access relaxes this tension, yet migrating execution adds transfer work and is harmful when the cluster is already congested. Thus, deciding where to execute a request requires reasoning jointly about cache locality, queueing, and the current operating point of the cluster. We present Vesta, a queueing-aware scheduler for distributed agentic LLM serving. Vesta assigns each session a stable holder for its authoritative KV cache while independently selecting an executor for every request. Grounded in classical queueing theory, its objective accounts for backlog, reusable prefixes, KV cache transfer, and the queueing externality of additional service demand; it migrates execution to recover idle capacity when beneficial and favors placements that add the least prefill and transfer work as congestion rises. Vesta also migrates reusable prefixes across length buckets to avoid cold starts. On real-world agentic workloads, Vesta improves throughput by 14.0\% and reduces mean TTFT by 57.8\% over the state-of-the-art scheduler in a PD-colocated deployment. In PD-disaggregated deployments, it reduces mean TTFT by up to 18.1\% and P99 TTFT by up to 29.0\% at matched throughput.

0.15

Balancing Workload Performance and Slurm Stress: Four Nextflow Deployment Strategies

cs.DC ·2026-08-13 ·Nil Tianchen Mu, William Dizon, Glen Otero, Torey Battelle
benchmark similarity 0.65
Abstract

Wide Nextflow fan-outs on shared Slurm clusters can submit tens of thousands of short tasks. Deployment settings route them through individual jobs, arrays, or nested schedulers inside enclosing allocations. These settings determine workflow turnaround and RPC volume, a shared cost that can degrade scheduler responsiveness. Existing comparisons evaluate whole workflow systems, while per-task queueing metrics cannot span architectures that dispatch inside existing allocations. We contribute a reproducible measurement protocol and benchmark harness. A clean-start clock begins before backend startup or allocation requests, placing architecturally different backends on a common time axis. Per-user Slurm sdiag counters attribute request count as the primary RPC demand measure and controller processing time as sensitivity context, separate from cluster-wide state. We apply the method to Slurm native dispatch, Slurm job arrays, HyperQueue, and Flux on the shared ASU Phoenix production cluster and a single-user Dev cluster. On Phoenix, every aggregation strategy improves both objectives relative to native dispatch; Flux has the lowest RPC demand, while HyperQueue's fastest median walltime is not stable across replicates. On Dev, arrays and Flux improve walltime, while HyperQueue trades the lowest RPC demand for the slowest completion. The method lets an HPC site compare deployment strategies using both user-visible performance and scheduler impact, then select the fastest strategy within its own RPC-demand limit.

0.04

Tight Worst-Case Completion-Time Bounds for Dominant Resource Fairness

cs.DS ·2026-11-05 ·Mugen Blue, Sungjin Im, Alexander Lindermayr
theory similarity 0.64
Abstract

Dominant Resource Fairness (DRF) is an influential policy for sharing multiple resources among jobs with heterogeneous demands. It combines sharing incentive, strategy-proofness, envy-freeness, and Pareto efficiency, making it particularly attractive for shared computing systems. DRF has been implemented in major resource-management frameworks, including Apache Mesos and Hadoop YARN. DRF is also nonclairvoyant: it requires no advance knowledge of job sizes, making it practical when processing requirements are difficult to predict. This motivates understanding its completion-time performance relative to an optimal schedule that knows all job sizes. However, elementary arguments can only give an $Ω(\sqrt D)$ lower bound and an $O(D)$ upper bound for total completion time, where $D$ is the number of resource types. In this paper, we close this gap by proving that DRF is $Θ(\sqrt D)$-competitive. More generally, for every fixed real $q\geq1$, we show that DRF is $Θ(D^{1/(q+1)})$-competitive for the $\ell_q$ norm of completion times, with larger $q$ placing greater emphasis on late-finishing jobs. These bounds hold with arbitrary release times. We also give a refined analysis of total completion time for small $D$.

0.17

Holistic Buoyancy-Driven Cloud Native Resource Management

cs.DC ·2026-10-19 ·Oliver Larsson, Thijs Metsch, Cristian Klein, Erik Elmroth
similarity 0.64
Abstract · 12 pages, 8 figures, 3 tables

Efficient resource management is a challenge in multi-tenant cloud environments as workloads compete for shared resources such as last-level cache and memory bandwidth, whose congestion can change dynamically as workload composition and resource demands evolve. Existing resource management approaches often rely on static resource requirements or prior workload profiling, limiting their ability to respond to changing resource contention as required in the cloud. In this paper, we leverage workload buoyancy, a runtime framework for capturing workload sensitivity to individual resources and application-level performance, to enable adaptive scheduling and resource management. We propose a holistic approach that incorporates buoyancy into workload scheduling, node-local resource allocation, and workload rescheduling throughout the workload lifecycle. We implement our approach as a Kubernetes-native prototype and evaluate it on a multi-node cluster using diverse cloud workloads under varying levels of resource contention. In a high-congestion scenario, our approach increases the percentage of workloads satisfying their SLOs from 7.74% to 87.8% on average compared with the Kubernetes baseline, while reducing resource congestion, the severity of SLO violations, and cluster fragmentation using limited rescheduling.

0.23

Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving

cs.DC cs.LG ·2026-09-16 ·Dev Bali, Soujanya Ponnapalli, Yichuan Wang, Natacha Crooks, Scott Shenker +1 more
similarity 0.63
Abstract

LLM serving is typically offered as a shared, multi-tenant service, where high-demand workloads from one client can cause latency SLO violations for others. Existing solutions for performance isolation equalize client throughput in the long run, for example through queueing and batching fairness. However, these approaches do not provide latency isolation guarantees; as a result, well-behaved clients can still experience significant degradation to their token-level latencies. In this paper, we present FairInference, which provides the novel δ-token fairness guarantee: for a well-behaved client, if a token is generated in d time units in isolation, it will be generated within d + δ time units in multi-tenant execution, providing strong latency isolation guarantees for LLM serving. To achieve this, FairInference addresses a key challenge of LLM serving: bounding delays from sharing GPU resources without support for fine-grained scheduling or resource allocation. In FairInference, the scheduler enforces per-token deadlines, while bounding the delays from GPU compute sharing and accounting for the additional delays introduced by the shared KV caching in GPU memory. We show that FairInference effectively bounds token-level latency spikes for well-behaved clients and improves overall throughput compared to state-of-the-art LLM serving systems.

0.46

List Update with Per-Page Maximum Delay

cs.DS ·2026-11-10 ·Tianhang Lu, Runtian Ren, Shengcai Liu
similarity 0.63
Abstract

We study list update with per-page maximum delay. Requests may wait, and one access clears all pending requests of a page at its current list rank, paying their maximum waiting penalty. Each service can change the permutation and therefore the cost of later accesses. We give a strictly $5$-competitive deterministic algorithm: serve a page when the oldest waiting time reaches two thirds of its current rank, then move it to the front. Our main tool is an asynchronous marked-rank coupling that compares different list orders and different service histories. A temporal transfer principle separates this permutation argument from two waiting-cost inequalities. It gives factor $5$ for page-specific superadditive penalties, common calendar rates, and additional hard deadlines; deadlines alone admit factor $3$. Two structural conditions on request-specific penalties give factor $3+2κ$. In particular, nonincreasing request severity retains factor $5$ even with unbounded weight ratios. For linear delay, offline optimization is NP-complete but fixed-parameter tractable in the number of pages. We characterize fixed-list batching, show an $Ω(m)$ gap between the best static order and dynamic reordering, and prove that the optimal deterministic factor for one page is exactly $2$. Counterexamples delineate the threshold rule's scope.

0.20