CS Paper Radar

Orrery: Modeling and Simulation of GPU-direct MPI Communication with SimGrid

Ahmad Moh'd Saleh A. Belbeisi, Amir Raoofy, Josef Weidendorfer, Majid Salimi Beni

cs.DC cs.PF ·submitted 2026-10-13 15:33 UTC · v1
published

Abstract

Simulation of HPC applications is an essential tool for analyzing system design trade-offs and enabling effective hardware--software co-design. As modern supercomputers increasingly rely on GPU acceleration, MPI implementations have evolved to support GPU-aware communication, enabling direct GPU data transfers. Modeling these mechanisms is therefore critical for simulating HPC applications on current systems. In this paper, we introduce Orrery, an interposition layer built on SimGrid's SMPI to model GPU-aware MPI communication using existing collective implementations and flow-based simulation. Orrery intercepts GPU memory allocations, kernel launches, data transfers, and MPI operations, and models their execution in SimGrid according to target system specifications, enabling simulation without requiring physical GPUs. We validate Orrery using CUDA-aware MPI microbenchmarks and a Jacobi solver on up to 32 nodes of the MUSICA supercomputer and demonstrate that it reproduces key GPU-aware MPI mechanisms.

Comment
Accepted as a short paper at the PMBS 2026 workshop (held in conjunction with SC26)

Closest papers

Nearest neighbours by meaning across the whole index.

Portability of Fortran's 'do concurrent' on GPUs II

cs.PL astro-ph.IMastro-ph.SRcs.CE+1 ·2026-08-20 ·Ronald M. Caplan, Miko M. Stulajter, Jon A. Linker, Jeff Larkin, Nikolaos Tselepidis +4 more
similarity 0.70
Abstract · 12 pages, 6 figures

There continues to be growing interest in using standard language constructs for parallel and accelerated HPC computing, avoiding the need for (sometimes vendor-specific) external APIs. For Fortran applications, language features such as 'do concurrent' loops open the door for compilers to implement multi-threaded, GPU-accelerated, and even distributed multi-node code with only the standard language. Here, we explore the current status of using 'do concurrent' for GPU-accelerated Fortran applications across three major GPU vendors (NVIDIA, AMD, and Intel). Using a production application, we test their current capabilities, showing where the standard language alone can be used, and where augmenting the code with a directive-based API (e.g., OpenMP) is still desirable or required. Multi-GPU tests are performed with GPU-aware MPI libraries. We find that the three GPU vendors can now GPU-accelerate pure Fortran (zero directives), but that manual data movement directives can help with performance and compatibility. The results show that there is rapid advancement towards making GPU-accelerated scientific HPC code performance portable using the Fortran standard language.

0.09

Validation-Centric AI-Assisted GPU Porting of a 250,000+ Line Legacy Weather Simulation Code

cs.DC ·2026-08-13 ·Tetsuya Hoshino, Masaya Kato, Kazuhisa Tsuboki, Daichi Mukunoki, Takahiro Katagiri +1 more
similarity 0.70
Abstract · 11 pages, 1 figure, 3 tables. Submitted to AgenticAI4HPC 2026

Recent advances in large language models have made CLI-based AI agents a practical tool for accelerating GPU porting of large legacy scientific applications. Such applications, however, are not merely old code bases; they are scientific assets whose credibility has been accumulated through long-term development, comparison with observations, and use in domain studies. GPU porting must therefore preserve this scientific validity while adapting the implementation to GPU-centric HPC systems. This paper presents a validation-centric AI-assisted GPU porting workflow through a case study of CReSS, a legacy Fortran weather simulation code with more than 250,000 lines. The workflow uses an AI agent to extract OpenMP regions, generate dump-based kernel benchmarks from physically meaningful simulation states, apply OpenACC transformations, and validate results through element-wise comparison with dumped reference data and application-level validation. Using a real typhoon simulation, the workflow produced numerically validated GPU implementations for 162 target kernels and achieved a 5.1x application-level speedup within practical wall-clock development cost. In particular, it detected numerical discrepancies in five kernels caused by floating-point and intrinsic-function differences, including threshold-sensitive branch divergence and cancellation effects, enabling feedback to the application developers. The case study suggests that, for large legacy scientific applications requiring dump-based validation, practical AI-assisted GPU porting must manage session-spanning context, runtime-state reconstruction, and costly recovery from small static-analysis omissions. These findings demonstrate that AI-assisted GPU porting requires not only code generation, but validation-centric workflow design.

0.12

FlashGPU-sim: Enabling GPU Modeling for Modern Architectures and AI Workloads

cs.AR cs.DC ·2026-09-14 ·Siying Yu, Yixun Hong, Guozhi Qiu, Jingci Liu, Feng Gu +4 more
top venuecode/data similarity 0.69
Abstract · To appear in MICRO 2026

As AI becomes increasingly ubiquitous, modern AI systems are shaped by a tight software-hardware co-design loop. Later GPUs expose features such as asynchronous data movement, tensor core pipelines, and fine-grained synchronization that high-performance kernels aggressively exploit, while emerging application behaviors increasingly influence the next generation of hardware design. Unfortunately, the latest open-source simulators for NVIDIA GPUs focus on architectures and software stacks from roughly six years ago. Therefore, they cannot support many state-of-the-art AI kernels generated by modern compiler stacks, e.g. Triton, or accurately model the hardware features they depend on. As a result, architects lack a credible platform for analyzing bottlenecks in this flywheel or evaluating design trade-offs for future AI systems. To bridge this gap, we present FlashGPU-sim, an open-source, execution-driven, cycle-accurate GPU simulator for modern AI workloads. FlashGPU-sim faithfully models modern hardware features such as asynchronous data movement, fine-grained synchronization, tensor-core execution, and distributed shared memory. A Triton extraction front-end allows direct simulation of optimized AI operators without manual porting, while multi-threaded execution makes large-scale software-hardware co-design practical. Across 131 workload configurations on RTX 5090, H100, and B200, FlashGPU-sim achieves a cycle-level MAPE of 5.24%, while multi-threaded simulation reaches a 7.86x speedup with 16 host threads. An H100 case study further demonstrates its utility for microarchitectural design exploration.

0.44

Uncovering the Memory Hierarchy of Modern NVIDIA GPUs for Accurate Simulation

cs.AR ·2026-11-10 ·Ziwen Wang, Jiapeng Zhou, Changxi Liu, Dongsheng Wang, Yun Chen
similarity 0.69
Abstract · 14 pages, 8 figures, 6 tables

Emerging artificial intelligence workloads, particularly Large Language Models (LLMs), place unprecedented demands on GPU memory systems. Consequently, architectural innovations increasingly target the memory system to alleviate these bottlenecks. However, existing works still rely on outdated memory hierarchy models that fail to capture the complexities of modern hardware. This fidelity gap leads to inaccurate characterization and could misguide optimization strategies. To bridge this gap, we present GemAtlas, the first systematic reverse engineering of the memory hierarchy in modern NVIDIA GPUs. By developing fine-grained SASS-level benchmarking, we expose critical, previously undocumented micro-architectural details across different cache levels, including cache set associativity, slice-routing and set-indexing hash functions, replacement policies, coherence behavior, and interconnect topologies. We integrate these discovered micro-architecture details into the state-of-the-art GPU simulation framework. The final results demonstrate that GemAtlas reduces the mean absolute percentage error (MAPE) from 31.69% to 22.24% on average across four GPUs. Specifically, GemAtlas reduces the MAPE from 34.04% to 23.98% for real-world LLM operators.

0.25

Energy-Aware Performance Characterization of Kokkos Kernels Across Precision Levels

cs.PF ·2026-10-22 ·Maria Patrou, Andrey Prokopenko, Aditya Kashi, Oscar Hernandez, Jean-Luc Fattebert
publishedbenchmark similarity 0.68
Abstract · 13 pages, 4 figures, Accepted at the 3rd International Workshop on Energy Efficiency with Sustainable Performance: Techniques, Tools, and Be…

Kokkos is a performance-portable programming model used in GPU-accelerated HPC applications comprising kernels with diverse computational and memory behaviors. We characterize kernels from two applications to evaluate how FP32, FP64, and mixed FP32/FP64 affect arithmetic intensity, performance throughput, and GPU energy consumption. We introduce an Energy-Aware Roofline Model that incorporates energy into performance analysis and a Thermal-Aware Inter-Kernel Energy Model that visualizes energy consumption alongside duration, GPU temperature, and execution order. We study both applications with: (1) fine-grained, application-level instrumentation through the Kokkos Performance Tools PAPI Connector and (2) coarse-grained PAPI instrumentation applied to a dedicated-kernel benchmark using vendor-specific power management components. Our analysis shows that precision effects vary by application, kernel, and workload. Mixed FP32/FP64 kernels generally achieve arithmetic intensity comparable to FP32, with modestly lower throughput and, in most cases, a modest increase in energy consumption. FP64 kernels generally exhibit the highest energy consumption among the evaluated precision configurations.

0.35

Architecting the Next Generation of Asynchronous, Distributed GPUs for the AI Era

cs.AR ·2026-08-23 ·Junrui Pan, Weili An, Cesar Avalos Baddouh, Christin David Bose, Ni Kang +7 more
highly cited similarity 0.68
Abstract

The rapid evolution of machine learning workloads has fundamentally transformed GPU hardware, driving architectures toward Multi-Chip Module (MCM) topologies, asynchronous execution primitives, and persistent, multi-phase kernel behaviors. Despite these shifts, cycle-level simulation infrastructure has lagged behind, lacking the native capability to model the physical non-uniformity of modern GPUs alongside the massive scale of state-of-the-art AI workloads. To bridge this gap, we present a cycle-level simulation framework designed to accurately model modern GPU generations, including Ampere, Hopper, and Blackwell. Rigorously validated against physical silicon, the simulator achieves a 99% Pearson correlation coefficient and a 13.4% mean absolute cycle error on the H100 GPU. Utilizing this infrastructure, we conduct architectural case studies to evaluate emerging design trajectories, including chiplet topology scaling, expanded SRAM capacity and bandwidth, and inter-GPU prefetching strategies.

0.39

Evaluating OpenMP Offloading for Intra-node Multi-GPU Programming across NVIDIA, AMD, and Intel Architectures: A 3D Heat Transfer Case Study

cs.DC ·2026-08-12 ·Ezhilmathi Krishnasamy
published similarity 0.67
Abstract

Currently, most supercomputers are equipped with GPUs from manufacturers such as NVIDIA, AMD, or Intel, which provide substantial parallelism and high throughput. It is common for a single compute node (intranode) to host multiple GPUs, typically four or more. Therefore, effectively leveraging all these GPUs within a single compute node is essential for applications in scientific and engineering domains. However, several factors must be considered before utilizing these GPUs for scientific computing, including the implementation of data communication, the programming models available for use across these GPUs, and the level of performance that can be achieved with a single codebase across different GPU architectures and configurations within a single compute node. OpenMP Offloading is a prominent directive-based programming model that can be executed on all three GPU types: NVIDIA, AMD, and Intel. In this research, we present an analysis of the benefits and performance challenges of using OpenMP Offloading to address the 3D heat equations, which involve both primary computation, as well as halo computation and communication. For additional comparison and scalability study, we also consider the Conjugate Gradient method. We investigate how performance varies in relation to native GPU programming models-CUDA for NVIDIA, HIP for AMD, and SYCL for Intel. Furthermore, we demonstrate that OpenMP Offloading can achieve performance improvements of approximately 2x for 2 GPUs and around 4x for 4 GPUs when compared to single-GPU OpenMP Offloading implementations across all three GPU types. This analysis is conducted systematically through various OpenMP Offloading implementations that utilize different low-level APIs for memory allocation, memory transfer options (synchronous, asynchronous, and peer-to-peer), and other native GPU programming models such as CUDA(NVIDIA),HIP(AMD),and SYCL(Intel).

0.22

Multi-GPU Communication Schemes for Iterative Solvers: When CPUs are Not in Charge

cs.DC ·2026-10-11 ·Ismayil Ismayilov, Javid Baydamirli, Doğan Sağbili, Mohamed Wahib, Didem Unat
top venuepositioncode/datahighly cited similarity 0.66
Abstract

This paper proposes a fully autonomous execution model for multi-GPU applications that completely excludes the involvement of the CPU beyond the initial kernel launch. In a typical multi-GPU application, the host serves as the orchestrator of execution by directly launching kernels, issuing communication calls, and acting as a synchronizer for devices. We argue that this orchestration, or control flow path, causes undue overhead and can be delegated entirely to devices to improve performance in applications that require communication among peers. For the proposed CPU-free execution model, we leverage existing techniques such as persistent kernels, thread block specialization, device-side barriers, and device-initiated communication routines to write fully autonomous multi-GPU code and achieve significantly reduced communication overheads. We demonstrate our proposed model on two broadly used iterative solvers, 2D/3D Jacobi stencil and Conjugate Gradient(CG). Compared to the CPU-controlled baselines, the CPU-free model can improve 3D stencil communication latency by 58.8% and provide a 1.63x speedup for CG on 8 NVIDIA A100 GPUs. The project code is available at https://github.com/ParCoreLab/CPU-Free-model.

0.71