CS Paper Radar

Characterizing Performance Tradeoffs in Heterogeneous GPU-Wafer-Scale Accelerator Systems

Ziyu Hu, Zhijing Ye, Jiamin Wang, Zhiqing Zhong, Zhendong Mi, Shaoyi Huang, Weijian Zheng, Rajkumar Kettimuthu, Xiaodong Yu

cs.DC cs.AR ·submitted 2026-10-29 02:42 UTC · v2

Abstract

As AI accelerator architectures become increasingly diverse, integrating different architectures into a unified system is emerging as a new heterogeneous computing paradigm. GPUs and WSEs differ substantially in compute, memory, and communication organization, creating both architectural complementarity and additional overheads, yet how to translate this heterogeneity into practical performance gains remains unclear. To address this gap, we propose a hierarchical benchmarking methodology to characterize GPU--WSE heterogeneous execution and the conditions for exploiting complementarity, together with an analytical performance model that identifies when heterogeneity becomes a bottleneck and when it yields system-level performance gains.

Closest papers

Nearest neighbours by meaning across the whole index.

Operating System Support for Accelerators as First-Class Compute Elements in Heterogeneous Systems

cs.AR cs.DCcs.OS ·2026-10-24 ·Mohammad Shahidzadeh, Lesley Shannon
similarity 0.69
Abstract · 13 pages, 12 figures

Heterogeneous systems-on-chip (SoCs) increasingly rely on domain-specific hardware accelerators (HAs) to deliver performance and energy efficiency gains beyond what general-purpose processors can provide. However, operating systems (O/S), such as Linux, assume that processors are the Compute Elements (CEs) and HAs are peripheral devices rather than specialized CEs. As a result, HA execution incurs substantial software (SW) overheads that reduce the effective benefits of hardware (HW) acceleration. This work demonstrates that significant inefficiencies arise from ``O/S peripheral device taxes'' (O/S taxes), including SW stack-accelerator communication, invocation, interrupt handling, and scheduling. We built MEERKAT, a Field-Programmable Gate Array (FPGA)-based framework for configurable multicore systems that include HAs; it allows the user to migrate Linux O/S support from SW to HW and identify, study, and quantify the impact. Here we study four major sources of O/S tax and introduce techniques to mitigate them. These include a virtual-memory-aware (VMA) communication method that enables efficient HA access to application data, a lightweight HA invocation interface, Interrupt Policy Adaptation (IPA) for dynamic polling and interrupt selection, and an HA-aware scheduler that dynamically distributes work across Central Processing Units (CPUs) and multiple HA instances. Experimental results demonstrate up to $260\times$ faster communication and $2$--$6\times$ performance improvements in wall-clock time over conventional schedulers. Furthermore, our study reveals that overall system performance depends not only on HA speed, but also on O/S overheads, resource availability, and scheduling decisions. These findings highlight the importance of treating HAs as first-class computational resources and motivate a more accelerator-centric O/S design for future heterogeneous systems.

0.15

RapidMoE: Exploiting Cross-Asymmetry via Adaptive Residual Offloading for Large-Scale MoE Inference

cs.DC ·2026-10-01 ·Wenxun Wang, Likai Ma, Zongle Huang, Chen Tang, Yongpan Liu
published similarity 0.69
Abstract · Accepted by EuroSys 2027. 17 pages

The widespread adoption of Mixture-of-Experts (MoE) has created a growing need for deployment on heterogeneous platforms. However, it exposes a fundamental mismatch between the algorithmic demands of large-scale MoE and the disparate characteristics of hardware.Existing CPU-GPU hybrid inference systems fail to resolve this as they either encounter PCIe bandwidth bottlenecks when loading experts to GPUs, or rely heavily on CPU computation. Consequently, this leads to low resource utilization and inevitable violations of fixed latency budgets as parameters scale. In this paper, we identify and exploit Cross-Asymmetry--a structural alignment between the algorithmic workload skew of MoE routing and the physical disparity of heterogeneous hardware. To this end, we introduce RapidMoE, a residual offloading system for efficient large-scale MoE inference. We propose how RapidMoE leverages a residual-split framework to enable offloading paradigm shift from expert-level to bit-level, which unfolds across three key dimensions: (1) data representation, enabling compact and decoupled storage; (2) routing strategy, partitioning computation into dual paths aligned with hardware capabilities; (3) execution parallelism, scheduling a balanced storage-compute workload across devices. We further employ a novel Unified Multi-Level Importance Arbitration to adaptively adjust the critical expert set at runtime, ensuring the accuracy-latency Pareto frontier. These innovations exploit inherent cross-asymmetry, fundamentally breaking the algorithm-hardware misalignment. Experimental results show that RapidMoE achieves up to 3.5x speedup in decoding and 2.1x speedup in prefill compared to state-of-the-art (SOTA) offloading systems.

0.60

Provisioning Networks for AI Supercomputers: A Trace-Driven Study of Performance Sensitivity at Unprecedented Scale

cs.NI ·2026-10-30 ·Tommaso Bonato, Amirreza Rastegari, Daniele De Sensi, Timo Schneider, Siyuan Shen +3 more
publishedtheory similarity 0.69
Abstract · To appear in the proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC 2026)

AI workloads on supercomputers make interconnect performance a primary design constraint. Hardware specifications and microbenchmarks alone do not show how real workloads respond to latency, bandwidth, and jitter. We present a trace-driven study of network sensitivity for training and inference workloads at scales up to 4,096 GPUs. Our methodology combines signature-based trace decomposition with linear programming to make broad network-parameter sweeps feasible. It preserves the recorded NCCL schedules and program dependencies, and estimates a lower bound on runtime under the LogGP model. We validate the compositional analysis against monolithic linear programs, LogGOPSim, and hardware measurements at feasible scales. The resulting latency-bandwidth envelopes identify low-sensitivity regions and performance cliffs that vary substantially across workloads. They also show when bandwidth upgrades provide meaningful returns and when additional network performance has little effect. These results translate execution traces into practical guidance for evaluating network configurations while making the model assumptions explicit.

0.39

Balancing Generality and Specialization: A Survey on AI Datacenter Hardware Architecture

cs.AR ·2026-09-21 ·Yufeng Gu, Jiazhen Wang, Reetuparna Das
surveycode/data similarity 0.68
Abstract

Rapidly growing AI workloads are driving large investments in AI datacenters. This survey classifies industrial AI accelerators into four architectural categories and compares their compute and memory organizations. It examines how node-, rack-, and pod-scale interconnects support collective communication, and traces architectural evolution across accelerator generations. The analysis connects advances in arithmetic throughput with changes in precision, data delivery, execution coordination, communication, power delivery, and cooling. It also discusses future design challenges arising from workload diversity, data movement, infrastructure constraints, and model evolution, showing how the trade-off between generality and specialization extends from individual accelerators to datacenter-scale systems. GitHub: github.com/Yufeng98/AI-datacenter

0.21

HeteroTask: Multi-Variant Task Scheduling and Execution on Distributed Heterogeneous Clusters

cs.DC ·2026-11-05 ·Hongzheng Tian, Alok Mishra, Haocheng Xu, Dejan Milojicic, Sitao Huang
benchmark similarity 0.67
Abstract · 11 pages, 8 figures

Heterogeneous systems that combine CPUs and GPUs across multiple servers offer substantial compute capability but complicate scheduling. For DAG-based applications with multiple task implementations, effective execution requires coordinating variant selection, device placement, and data movement. Offline planning can exploit global dependency information, yet plans derived from isolated kernel profiles may diverge from execution because of runtime overheads. We introduce HeteroTask, a framework integrating kernel-wise profiling, PEFT-based multi-variant planning, host-buffer lifetime modeling, and schedule-driven MPI execution. A Bayesian-optimization loop calibrates empirical overhead corrections for repeated, fixed-DAG workflows on exclusively allocated nodes. We evaluate five benchmark and workflow-derived applications on up to six 8-GPU nodes. On two 8-GPU nodes, HeteroTask reduces application-level completion time by 19.7-72.1% relative to the compared StarPU implementations. For Canny Edge Detection, calibration reduces task-window makespan by 25.0% relative to planning without overhead corrections. Combined with CPU/GPU variant selection, calibration reduces it by 58.9% relative to an uncalibrated GPU-only execution.

0.24

Performance Portability of Out-of-Kernel Optimizations on GPUs: A Molecular Docking Case Study

cs.DC cs.PF ·2026-11-04 ·Gianmarco Accordi, Leonardo Beltrame, Theresa Pollinger, Jens Domke, Davide Gadioli +1 more
similarity 0.67
Abstract · 11 pages, 5 figures, 1 table

Achieving performance portability is challenging for irregular scientific applications built as repeated multi-kernel GPU pipelines, because throughput depends on out-of-kernel decisions as much as on kernel efficiency. Using molecular docking as a representative case study, we propose an out-of-kernel tuning methodology that treats batching policy, worker concurrency, and per-worker memory allocation as explicit optimization parameters, replacing backend-dependent occupancy-driven choices with a more portable execution strategy based on explicit resource bounds. We implement this methodology in the muDock mini-app under a manager/worker execution model where multiple host workers share a GPU through independent streams. We then evaluate the methodology on NVIDIA, AMD, and Intel GPUs across CUDA, HIP, and SYCL implementations. Across all evaluated platform/backend pairs, the memory-bounded formulation exceeds the peak throughput of the occupancy-guided baseline. The fully tuned methodology improves the end-to-end throughput on all tested configurations, achieving an average speedup of 1.53x over the baseline.

0.25

Multi-GPU MoE Inference with Adaptive Expert Offloading

cs.AI ·2026-11-06 ·William X. Zheng, Zhongjie Chen, Shuo Yang, Ziming Mao, Xiaoze Fan +7 more
similarity 0.67
Abstract · 15 pages, including references and appendix

Modern Mixture-of-Experts (MoE) inference is constrained by GPU memory. This constraint is mitigated by expert offloading, which stores part of the model's weight in host memory and executes on demand. Though MoE architecture has computation sparsity, more GPUs are needed for serving increasingly larger frontier models. However, existing expert offloading systems overlook this need and fail to scale performance with additional GPUs. We present AdaOffload, a general and efficient multi-GPU hybrid expert execution framework that effectively handles heterogeneous hardware and workloads. We co-design GPU-CPU parallelism and GPU-CPU co-execution for decode and prefill with online workload observation that adapts to different hardware, workload, and batch size heterogeneity. We evaluated AdaOffload on five different platforms with varying GPU capability (L4, Pro6000, B200), number of GPUs, CPU capability, and NUMA topology on three different frontier open-weight models (284B to 753B). AdaOffload achieves $1.9\times$ to $15.5\times$ throughput speedups across different models over state-of-the-art systems.

0.21

GPU implementation of a resource-constrained virtual machine

cs.DC cs.GR ·2026-08-17 ·Simone Li, Vladislav Brusokas, Andrei Ghita, Shuxuan Li, Wim Vanderbauwhede
benchmark similarity 0.67
Abstract · 5 pages, 5 figures. Accepted at the 2nd International Workshop on Low Carbon Computing (LOCO 2026), Lancaster University, United Kingdom, 10…

One of the main reasons compute hardware becomes obsolete is software bloat: resource requirements increase for every iteration of a software product. Resource constrained VMs are one way to combat software bloat as they post a hard limit on the resources and so force the programmer to be frugal. In this paper we explore the deployment of one such resource constrained VM, Uxn, on GPU. We show that for competitive performance it is essential to make use of the GPU data parallelism. We present an OpenMP-style parallelism API for Uxntal, the stack-based assembly-style language for the Uxn platform. We demonstrate that exemplar code using our API can run at comparable performance even on an integrated GPU. Specifically, our evaluation results show that using this approach improves performance on the compute-intensive Stencil benchmark with 19x and frame rate on the graphics-intensive Bunnymark benchmark with 7x. In practice, all laptops and desktops and even mobile devices have a GPU and our work shows that they can be used to execute frugal workloads effectively.

0.25