CS Paper Radar

Optimal Scheduling in Generalized Switch in Heavy Traffic

Runhan Xie, Ziv Scully, Rhonda Righter, Izzy Grosof

cs.PF ·submitted 2026-09-15 18:42 UTC · v1
theory

Abstract

The generalized switch is a highly flexible queueing model, covering multiclass, multiserver, and multiresource queueing systems as well as a wide variety of stochastic networks. Although many scheduling policies have been developed for this model, they almost entirely address unknown job duration settings. How to optimally use known job durations in the generalized switch has remained open. Moreover, optimizing mean response time remains open in both settings. We introduce the first policy to guarantee heavy-traffic optimal mean response time in the generalized switch, our Smallest Equalizing Bucket (SEB) policy. The key challenge in designing an optimal scheduling policy is that we must simultaneously prioritize small jobs and also minimize resource waste, all while fitting within the generalized switch's service options. SEB overcomes this challenge by grouping jobs into duration-based buckets and enforcing an "equalizing" service structure that keeps each bucket balanced while still prioritizing the smallest jobs. We prove SEB's heavy-traffic optimality. Simulations further confirm the effectiveness of SEB-inspired heuristics.

Closest papers

Nearest neighbours by meaning across the whole index.

Sparse Predict and Demote: Optimal Scheduling with Minimal Size Advice

cs.PF cs.DScs.LG ·2026-11-06 ·Chengyi Liu, Thodoris Lykouris, Junxuan Shen, Adam Wierman
similarity 0.64
Abstract

Knowledge of job sizes is crucial for optimizing the mean response time in single-server scheduling. However, predictions of job sizes are typically inexact and may be costly to obtain. Thus, a long-standing research question is: what is the minimal information about job sizes required to obtain near-optimal mean response time? We introduce a novel policy, \emph{Sparse Predict and Demote (SPD)}, and prove that it achieves order-optimal mean response time in heavy traffic while requiring only one-bit job-size predictions for a vanishing fraction of arrivals. SPD achieves heavy-traffic order-optimality even in situations when predictions are (i) inexact up to any fixed multiplicative error bound, and (ii) costly, i.e., the server is used to compute predictions and is thus unavailable to serve jobs while processing prediction queries. Finally, we use numerical experiments to illustrate that SPD maintains near-optimal performance beyond the heavy-traffic regime.

0.20

Ofan: Optimal Load Balancing for AI Training

cs.NI cs.DCcs.PF ·2026-10-05 ·Sarah McClure, Evyatar Cohen, Jakob Krebs, Alexander Shpiner, Mark Silberstein +3 more
theory similarity 0.63
Abstract · Extends the second part (Ofan) of arXiv:2507.21372

The extreme collective completion time (CCT) demands of AI workloads challenge existing packet spraying algorithms, which can have trouble efficiently load-balancing workloads that are sent at full line rates. We trace this to a structural cause: on a fat tree, once a packet picks its upward path, the downward path to its destination is unique, so destination-oblivious schemes cannot undo the imbalance it creates. We prove that such schemes can suffer from $Θ(\sqrt{m})$ queueing for messages of size $m$, thus eventually triggering rate reductions by the congestion control. Instead, we suggest Ofan, a switch-based destination-aware LB scheme that can reach $O(1)$ queueing. We also present its pOfan variant that fits the pipe-based architecture of current switches. Our P4 implementation shows that it consumes modest resources. An end-to-end FSDP2 evaluation with Llama-3 405B-parameter models shows that Ofan cuts CCT inflation by $16$--$39\times$ when compared to existing algorithms.

0.26

Approaching Optimal Scaling of Queue Size for an Input-Queued Switch

cs.PF math.PR ·2026-10-27 ·Taha Ameen, Weina Wang
similarity 0.62
Abstract

We study the scaling of the expected total queue length in an $n\times n$ input-queued switch. It has been conjectured that the optimal scaling is $O(n/(1-ρ))$, up to logarithmic factors, where $ρ$ is the system load. This conjecture has been established in the traditional heavy-traffic regime and in certain sub-regimes of the large-system regime. But it remains unresolved over a wide range of load scalings in the large-system regime. In this paper, we develop a family of scheduling policies parameterized by a positive integer $k\ge 2$. We show that under uniform traffic, our policy achieves an expected total queue length of $O\bigl({n\log\max\{n,1/(1-ρ)\}}\big/{(1-ρ)^{2^k/(2^k-1)}}\bigr)$, as long as $ρ\to 1$ in the large-system regime. As $k$ increases, the exponent $2^k/(2^k-1)$ approaches $1$, bringing our bound arbitrarily close to the conjectured scaling. The expected amortized scheduling cost of the policy is $O(n^3 k)$ per time slot. Our policy is of a batching type, with a novel recursive design that allows an arbitrary number of batching levels. This design principle may also be useful for other scheduling problems in which combinatorial constraints restrict which jobs can be served simultaneously.

0.20

Online Network Design with Maximum Delay and Stochastic Arrivals

cs.DS ·2026-11-10 ·Tianhang Lu
theory similarity 0.60
Abstract

We study online network services that combine pending requests into batches. Each batch pays for its spatial service and for the waiting time of its oldest request. The objective is the sum of these costs over all batches. Thus requests can share both a spatial purchase and one maximum-delay charge. We assume independent Poisson arrivals of known rates, while actual requests are revealed only upon arrival. For stateless services, suppose that each type has an attainable minimum cost when served alone, that every batch costs at least the stand-alone minimum of each type it contains, and that one finite-cost action can serve any batch, regardless of its request multiplicities. We give a deterministic algorithm with long-run ratio of expectations $e/(e-1)$ against an unrestricted offline optimum. This factor is optimal, even allowing randomization, already at one request location. The key comparison measures the union of time intervals associated with requests. The theorem applies directly to multi-root and pair aggregation on trees. We also study facility location, where each request pays its own connection distance and the bounded-action assumption can fail. A deterministic policy gives a ratio $3$ for arbitrary nonnegative opening costs. It immediately serves a finite initial set of requests, computes an assignment from their observed arrival gaps, and then keeps that assignment fixed. Its analysis combines Poisson insertion with a time-slice packing bound on the offline optimum. Every guarantee includes the full cost of completing finite inputs without an end-of-input signal.

0.15

Joint Service Depth and Priority Design in Queueing Systems

cs.GT econ.THmath.OC ·2026-10-23 ·Adam N. Elmachtoub, Yash Kanoria, Jonathan Y. Tan
similarity 0.59
Abstract

In AI-assisted service systems, the server can choose how much processing effort to spend on each job, which we call the service depth. Deeper reasoning improves output quality but occupies the server longer and increases congestion, and the realized processing time remains random because AI models are probabilistic and tasks vary in complexity. We study the joint design of service depth and priority in a single-server queue whose customers differ in delay sensitivity, with the goal of maximizing social welfare. For each customer type, the decision maker chooses a priority level and a service depth, and service times are exponential with mean equal to the chosen depth. With two customer types and linear service values, we characterize the centralized optimum. Counterintuitively, it prioritizes the patient type and gives it a smaller service depth when patient customers are sufficiently prevalent. When delay sensitivity is private, however, incentive compatibility requires the impatient type to receive a weakly shorter sojourn time, so the optimal price menu never prioritizes the patient type. Consequently, there can be a welfare gap between the centralized decision maker's optimal control policy and the optimal decentralized pricing mechanism. With a continuum of types, the centralized optimum is monotone: more delay-sensitive customers receive weakly smaller depth and higher priority, so the optimum can be implemented by a price menu with no welfare loss. Allowing randomized depths similarly closes the gap in the two-type model. Finally, we calibrate an agentic AI service system using published coding-agent trials from DeepSWE and numerically examine priority inversion, the value of joint design, and the welfare gain from personalization.

0.26

Queuing for Inference: Dynamic Memory Footprints and Costly Preemption

cs.PF cs.NI ·2026-10-24 ·Jiahui Ni, Vishrant Tripathi
similarity 0.58
Abstract

Modeling inference workloads for transformer-based neural networks poses unique challenges due to their dynamic memory footprints, head-of-line blocking caused by heavy-tailed request distributions, and memory-dependent preemption overhead arising from the need to load/unload cache state. In this paper, we introduce a new queuing model that captures the impacts of these features on system stability and scheduling design. We show that the Gittins-index policy, which minimizes mean response time in the standard M/G/1 setting, can incur substantial preemption overhead in our model. Motivated by this limitation, we propose the Multi-Level-Priority-Threshold (MLPT) policy, that limits preemption while preserving priority for short jobs. We characterize the mean response time under MLPT and provide empirical rules-of-thumb to solve the resulting threshold-optimization problem. Our key observation is that optimal scheduling is a balance between reducing the response times of shorter jobs and limiting the overhead incurred when preemption occurs. These results provide a theoretical framework for understanding scheduling in memory constrained queuing systems for inference tasks.

0.09

Stability of Fork-Join Systems with Redundancy and Heterogeneous Servers

cs.IT math.PR ·2026-09-08 ·Chutong Gao, Seyed Iravani, Ohad Perry
similarity 0.58
Abstract · 28 pages, 1 figure. Current status: Reject and Resubmit, Operations Research. Resubmitted on July 26, 2026

We consider the stability problem of fork-join systems with redundancy (FJR) and heterogeneous servers under both static and dynamic capacity-allocation policies. In an $(n,k)$ FJR system, each arriving job is split into $n$ independent tasks, with one task assigned to each of $n$ parallel servers. Once $k \le n$ tasks have been processed, they are joined and the corresponding job departs the system; the remaining $n-k$ unprocessed tasks are then removed and are therefore termed redundant. We first identify the nominal traffic intensity and characterize the maximal stability region, defined as the set of traffic intensities for which there exists an admissible policy that stabilizes the system. We then establish conditions under which this maximal stability region is attained for two classes of policies: static and dynamic. Specifically, we show that for static allocation policies, in which service capacities remain fixed over time, maximality is achieved whenever the fastest server is allocated no more than $1/k$ of the total service capacity. For dynamic allocation policies, in which a fixed total service capacity may be repeatedly reallocated among the servers, we show that maximality is achieved whenever the cumulative capacity allocated to the $j$ shortest queues does not exceed $j/k$ of the total capacity for every $j=1,\ldots,k-1$. Our analysis is based on a projection of the $(n,k)$ FJR system onto a simpler $(k,k)$ system that has no redundancy, together with a novel sample-path comparison argument for multidimensional processes based on the generalized Schur-convex order.

0.31

New Complexity Results for Fair Repetitive Scheduling

cs.DS ·2026-08-20 ·Moran Koren, Michael L. Pinedo, Dvir Shabtay
similarity 0.58
Abstract · 12 pages

We revisit the problem of finding fair solutions to repetitive scheduling problems with a single machine. In this problem, we are given a set of $n$ clients and a planning horizon consisting of $q$ periods (days). Each day, every client submits a single job that must be processed by the machine. The objective is to construct a set of $q$ schedules, one for each day, such that the quality of service (QoS) received by each client meets a predefined threshold. The QoS measure may be any standard scheduling criterion, such as the total waiting time or total completion time of a client's jobs over the entire planning horizon. This problem has been studied in the literature, with previous works providing complexity classifications and approximation algorithms for various QoS measures. Nevertheless, several important questions remain open. In this paper, we resolve three of these questions and identify several additional directions for future research.

0.23