CS Paper Radar

Performance Reporting of Mathematical Library Installations with LAAB - An Overview

Aravind Sankaran, Paolo Bientinesi

cs.PF cs.MS ·submitted 2026-08-13 17:35 UTC · v1
benchmarkcode/data

Abstract

We present the Linear Algebra Aware Benchmarks (LAAB) framework for systematically assessing and reporting the performance of mathematical library installations on HPC systems. Mathematical libraries provide interfaces for operations that form the computational building blocks of scientific applications. Reporting their performance is important for assessing application efficiency, estimating compute-time requirements, and preparing resource-allocation requests. In this paper, we define four objectives for performance reporting: 1) traceability, linking each report to the exact library installation and execution settings; 2) compatibility, relating library-operation performance to higher-level scientific applications that use them; 3) reliability, supporting interpretation in the presence of measurement variability; and 4) accessibility, ensuring that reports, benchmark definitions, and relevant metadata are available for inspection and reproduction. We then present the design of LAAB and show how it addresses the challenges associated with these objectives.

Comment
10 pages, 3 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Discovering Performance Archetypes: Critical-Path-Aware Pattern Analysis and Regression Detection

cs.PF ·2026-09-15 ·Kaveh Shahedi, Heng Li, Maxime Lamothe, Foutse Khomh
top venue similarity 0.57
Abstract

Software performance analysis and prediction requires integrating multiple signals, as code structure alone cannot capture runtime behavior shaped by execution frequency, resource contention, and I/O patterns. We present a critical-path-aware performance analysis methodology that automatically discovers recurring performance patterns by synthesizing static code features, dynamic execution traces, and kernel-level resource data. In a preliminary study across six real-world C/C++ applications (SQLite, OpenSSL, Zstandard, FFmpeg, cURL, and jq), we first empirically confirm that static complexity metrics explain only 10.4% of the variance ($ρ^2$) in critical path execution time, quantifying a gap that, while theoretically expected, had not been measured systematically across applications. Motivated by this finding, we analyze nearly 80,000 critical execution paths and address two research questions. First, we discover 13 distinct performance archetypes: recurring behavioral patterns that appear consistently across different applications, independent of their domain or implementation. Five of these patterns are near-universal and appear in at least five of the six applications studied. Notably, three of these archetypes are present in all six applications, and together, these common patterns account for 56.4% of all observed paths. Each archetype maps to specific resource profiles and optimization strategies that transfer across domains. Second, we leverage these archetypes within a multi-signal regression detection framework that triangulates path structure, resource consumption, and archetype deviations, achieving an F1-score of 0.867 and a 60.4% improvement over resource-only methods.

0.37

SysPilot: Cross-Library Performance Autotuning via Library Performance Contracts

cs.PF cs.AI ·2026-11-04 ·Yineng Yan, Haowei Chen, Christopher J. Rossbach
benchmarkcode/data similarity 0.57
Abstract · SysPilot is open source at https://github.com/yinengy/SysPilot

Modern scientific and machine-learning applications compose many independently developed libraries. When these libraries run together, their independently managed runtimes can oversubscribe shared resources: using the same CPU cores, fighting over GPUs, or contending for memory bandwidth. Avoiding such contention requires careful configuration across library boundaries. Existing autotuners do not directly investigate what resources are shared between libraries, which runtime signals correlate with interference. As a result, they are inefficient for finding optimal configurations to mitigate contention. This paper presents SysPilot, a cross-library performance autotuning system based on Library Performance Contract (LPC). A Library Performance Contract captures a library's resource demands, assumptions, controls, and observability signals in LPCL, a specification language for describing library resource requirements. SysPilot synthesizes contracts from documentation and code artifacts, composes contracts from the libraries used by an application, derives resource-conflict hypotheses, profiles the application to validate those hypotheses, and selects candidate configurations. This design makes the Library Performance Contract a reusable unit of performance knowledge: a contract for a library can be reused across many applications that depend on that library. We evaluate SysPilot on a benchmark suite of cross-library contention scenarios. The results show that SysPilot matches the performance of autotuners that rely on application-specific manual information while retaining cross-application reuse. Compared with general-purpose autotuners such as OpenTuner and SMAC3, SysPilot improves benchmark performance metrics by up to an additional 38% over the best configuration found by these autotuners, while using 3x fewer runs.

0.44

Performance Portability Explained: Insights from accelerating ICON microphysics

cs.DC cs.PFcs.PL ·2026-11-10 ·Faveo Hörold, Georgiana Mania, Panagiotis Adamidis, Harshada Balasubramanian, Jörg Behrens +13 more
similarity 0.56
Abstract

Scientific computing is under increasing pressure to adapt to HPC hardware that is shaped by AI workloads. In these modern ecosystems, extracting reasonable performance without investing major engineering effort is a central issue for domain scientists. Correctness constraints and intricate reference implementations represent further impediments to development. Instead of using a double-precision implementation as a reference, considering the underlying scientific problem directly opens more opportunities for performance improvements when updating and rewriting applications. We take seven different portability frameworks---covering pragma-based, DSL, and data-centric approaches---and evaluate them on a microphysics kernel of the ICON climate model. Our performance benchmarks show that each different HPC system benefits from a different combination of optimizations, so we develop the measure of framework expressivity to gauge how easily each framework can customize portable implementations in this way.

0.25

Productivity in HPC: A Critical Review from SLOC and Performance Portability to AI-Assisted Programming

cs.DC ·2026-10-05 ·Ami Marowka
similarity 0.55
Abstract · 39 pages, 2 figuers, 2 tables

Productivity has long been recognized as a central concern in high-performance computing (HPC), yet it still lacks a workable formal definition and a broadly accepted metric for quantified assessment. Recent attempts to incorporate productivity into performance-portability analysis have introduced code-divergence, convergence, and semantic measures, but have not established a convincing formal relationship between productivity and performance portability. More recent language and compiler systems, including JACC, Mojo, JLIR, and Accl++, pursue productive performance portability through higher-level abstractions, multi-level intermediate representations, automatic kernel generation, and runtime compilation. These systems demonstrate substantial engineering progress, while productivity is still commonly represented through programmability, ecosystem reuse, code size, or reduced implementation burden rather than direct measurement of human effort. This paper presents a critical review of HPC productivity measurement, tracing its development from source lines of code (SLOC), static complexity, and empirical usability studies to development-time models and semantic approaches. We also examine how automated migration and artificial-intelligence(AI)-assisted programming weaken the correspondence between code transformation and human effort, shifting effort toward verification, correction, and performance tuning. The reviewed methods are compared in terms of what they measure, directness, objectivity, reproducibility, practicality, and their relationship to performance and portability. We conclude by identifying the principal limitations and open challenges for quantified HPC productivity assessment.

0.34

Trustworthy Numerical Software in the Age of AI: What Would Turing Do?

cs.MS math.NA ·2026-11-16 ·Kimon Fountoulakis, Maksim Melnichenko, Muhammad Fetrat Qharabagh, Georgios Alexandros Georgantzas, George Spyrou +4 more
similarity 0.55
Abstract · 44 pages

AI coding agents can now produce numerical software faster than human experts can review it, raising serious concerns about usability and reliability. These and other AI tools are already transforming mathematics and software, and they will soon transform mathematical software. These concerns motivated us to assess AI-generated numerical linear algebra (NLA) kernels, which we did through an audit of a recent public GPU QR competition. All five leading kernels passed the competition's accuracy tests, and the winning kernel achieved a 107-fold speedup over the competition's baseline, which calls NVIDIA's QR routine. Yet each failed at least once on new inputs, even those from the competition's generator, under the same checker, while the baseline passed them all. In light of this, we want to develop reusable formalizations of numerical analysis methods, which remain sparsely represented in existing formal libraries. We describe VSCL, our Lean library of mathematical foundations for numerical computing. We also propose VERNA to connect these tasks by using formalized results to guide kernel assessment against explicit specifications. Drawing on Turing's work on rounding-error analysis and program checking, we discuss design principles for developing trustworthy numerical software when direct human checking is impractical.

0.14

Energy-Aware Performance Characterization of Kokkos Kernels Across Precision Levels

cs.PF ·2026-10-22 ·Maria Patrou, Andrey Prokopenko, Aditya Kashi, Oscar Hernandez, Jean-Luc Fattebert
publishedbenchmark similarity 0.55
Abstract · 13 pages, 4 figures, Accepted at the 3rd International Workshop on Energy Efficiency with Sustainable Performance: Techniques, Tools, and Be…

Kokkos is a performance-portable programming model used in GPU-accelerated HPC applications comprising kernels with diverse computational and memory behaviors. We characterize kernels from two applications to evaluate how FP32, FP64, and mixed FP32/FP64 affect arithmetic intensity, performance throughput, and GPU energy consumption. We introduce an Energy-Aware Roofline Model that incorporates energy into performance analysis and a Thermal-Aware Inter-Kernel Energy Model that visualizes energy consumption alongside duration, GPU temperature, and execution order. We study both applications with: (1) fine-grained, application-level instrumentation through the Kokkos Performance Tools PAPI Connector and (2) coarse-grained PAPI instrumentation applied to a dedicated-kernel benchmark using vendor-specific power management components. Our analysis shows that precision effects vary by application, kernel, and workload. Mixed FP32/FP64 kernels generally achieve arithmetic intensity comparable to FP32, with modestly lower throughput and, in most cases, a modest increase in energy consumption. FP64 kernels generally exhibit the highest energy consumption among the evaluated precision configurations.

0.35

PerfArena: Ranking LLMs by Code Performance

cs.PF cs.CLcs.SE ·2026-11-13 ·Rui Rua, Hailemariam Mersha, Joao Paulo Fernandes, Karim Ali
code/data similarity 0.54
Abstract

Industry now relies on LLMs to write a large share of production code. Although that code is mostly, but not always, correct, correctness alone does not determine whether it is fit for repeated and extensive use, which depends on what the code costs in time, memory and energy. Most code-generation benchmarks nonetheless evaluate correctness in isolation, and most performance-oriented benchmarks cover few languages and rarely measure energy consumption. To address this gap, we present PERFARENA, an open-source, reproducible arena that ranks LLMs by the performance of their generated code across multiple languages and competitions. PERFARENA executes the solutions of each LLM on a Unix-like host and reports correctness together with latency, peak memory, and energy consumption. Correctness oracles, performance oracles, profilers, and code-generation workflows are pluggable in PERFARENA, enabling the integration of new judging backends and measurement tools without changing the pipeline. We evaluate PERFARENA on two competitions: CLBGComp, 10 compute-intensive benchmarks from the Computer Language Benchmarks Game, and LeetCodeComp, 99 LeetCode problems whose accepted solutions differ measurably in latency. Currently, PERFARENA holds 9,455 correct and measured solutions from 13 state-of-the-art LLMs across 7 programming languages. Across both competitions, got-6-astra is the only LLM that no other LLM beats significantly in latency, energy consumption, or peak memory, solving 99.7% of LeetCodeComp and 100% of CLBGComp. All PERFARENA leaderboards and measured solutions are publicly available, providing a continuously updated reference that helps developers choose an LLM per language and metric, instead of relying only on correctness rates or pricing.

0.30

Characterizing the I/O Behavior of HPC Applications through Modeling and Simulation

cs.DC ·2026-08-27 ·Njoud O. Almaaitah, David E. Singh, Taylan Özden, Jesus Carretero, Raffaele Montella
hot direction similarity 0.53
Abstract

Parallel applications process large amounts of data, leading to intensive parallel I/O operations. These operations can exhibit different levels of complexity, including, among others, multiple I/O access patterns, data staging, and contention risks. Therefore, in order to exploit high-performance computing (HPC) systems efficiently and optimize the I/O performance, it is crucial to consider the I/O behaviour of the HPC applications. In this work, we have developed a framework that reproduces the I/O access pattern of real applications in a simulated environment provided by ElastiSim, a batch-system simulator for rigid, malleable, and evolving workloads. The simulated applications are generated based on I/O traces captured from real applications provided by the HPC Input/Output (HPCIO) analysis repository. The HPCIO analysis database includes traces combined with information about real applications' performance across different parallel I/O libraries and layers of the I/O stack. We have conducted detailed case studies of real-world applications' traces to demonstrate how the proposed modeling framework can provide insights into the performance characteristics of I/O applications, including the I/O congestion analysis based on the application's I/O access pattern.

0.23