CS Paper Radar

The Unseen Delta: Characterizing the Compiler Optimization Landscape via Top-Down Differential Analysis

Zhibo Liu, Huaijin Wang, Shuai Wang

cs.SE ·submitted 2026-08-10 12:32 UTC · v2
top venue

Abstract

Compiler optimizations are essential for achieving high performance in modern software. However, recent studies highlight the persistence of performance bugs, i.e., subtle defects where the compiler generates functionally correct but computationally inefficient code, leading to significant performance degradation. Existing detection and testing methods typically employ a bottom-up approach, focusing on specific low-level code properties and remaining confined to known optimization rules. Consequently, they struggle to quantify the holistic impact of identified issues and often overlook critical microarchitectural inefficiencies. We observe a key indicator of untapped potential: different compilers often produce binaries with significant performance differences for identical source code. However, the root causes of these discrepancies remain largely unexplored and difficult to pinpoint using current techniques. To bridge this gap, we introduce a top-down differential analysis methodology. This approach calibrates compiler optimization differences with fine-grained, hierarchical microarchitectural metrics, offering a comprehensive view of runtime behavior. Using a sampling-based approach, this method efficiently pinpoints the critical code snippets responsible for performance differences, enabling targeted root cause analysis. Our empirical evaluation uncovers substantial and often surprising performance differences between binaries generated by GCC and Clang. A categorization of root causes reveals systemic challenges in compiler optimizations. To quantitatively validate our findings and demonstrate practical impact, we developed a binary patching framework that fixes identified performance issues by transplanting superior code sequences from competing compilers. This work provides a novel lens for understanding and analyzing optimization defects.

Comment
To appear at ISSTA 2026; figures fixed
Journal ref
Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA '26), October 03--09, 2026, Oakland, California
DOI
10.1145/3832164

Closest papers

Nearest neighbours by meaning across the whole index.

Pattern-Driven Evaluation of LLM Code Optimization: Isolating Compiler-Transcendent Semantic Reasoning

cs.AI cs.PLcs.SE ·2026-10-17 ·Wesley Lu, Runzhou Liu, Hongru Du, Xinyao Yi
benchmark similarity 0.72
Abstract · 12 pages, 2 figures, 6 tables. Published at the AI Measurement Science Workshop at COLM 2026

Existing evaluations of LLM-based code optimization primarily measure end-to-end speedup, obscuring whether a model actually performs the intended optimization. For compiled languages, this is especially problematic: observed gains may reflect transformations already recoverable by modern optimizing compilers rather than optimization behavior attributable to the model itself. We present PatternBench, a benchmark for studying compiler-transcendent optimization, i.e., performance improvements that remain after aggressive compiler optimization. PatternBench targets residual optimization opportunities across seven classes of transformations and pairs runtime evaluation with an AST-based optimization faithfulness analysis that verifies whether the intended optimization is realized in the generated program. Experiments with frontier, open-weight, and fine-tuned models show that runtime speedup alone provides an incomplete account of optimization capability. Optimization faithfulness is strongly associated with realized speedup, and fine-tuning improves both faithfulness and optimization effectiveness. These results suggest that evaluation of LLM-based code optimization should measure not only performance outcomes, but also whether the target transformation was successfully executed.

0.19

XLA from the Trenches: Correctness on TPU Workloads at Scale

cs.AI cs.LGcs.PL ·2026-10-31 ·Ignacio Cano, Amir Saadat, Rishipal Singh Bhatia, Samuel Ginzburg, Patrick C. Toulme +17 more
similarity 0.72
Abstract

Accelerating modern foundational models has pushed domain-specific compilers toward increasingly complex, multi-tiered optimization pipelines. However, this heavy optimization focus introduces delicate tradeoffs between peak execution performance and correctness. Subtle codegen defects can trigger numerical degradation and memory or buffer corruption, invalidating multi-million-dollar training runs and degrading live serving quality without raising obvious compiler errors. In this paper, we present a battle-tested methodology for verifying and triaging Google's XLA:TPU compiler at hyperscale. We leverage fuzzing techniques, combining synthetic program generation with real production traces, while orchestrating AI agent swarms to autonomously generate stress tests, reproduce failures, and draft fixes. To pinpoint root causes in full workloads, we pair this with a hierarchical differential pipeline that bisects large graphs down to individual failing operations inside fused kernels. Deployed in production, our framework has already driven the resolution of over 142 compiler bugs (with 103 additional findings under review), from which we distill pragmatic lessons learned.

0.21

WINIC: An Automatic Cycle-Accurate Multi-Platform Microbenchmarking Tool for Instruction Performance Metrics

cs.PF ·2026-11-10 ·Tobias Rühl, Jan Laukemann, Georg Hager, Gerhard Wellein
code/data similarity 0.69
Abstract · 12 pages, 7 figures, 3 tables, 10 listings

Performance models allow developers to gain important insights into the interaction of their code with the hardware that executes it, and enable them to apply target-specific optimizations. To create in-core performance models for out-of-order processors, architecture-specific performance metrics like latency, throughput, and port binding of individual instructions are indispensable. For x86 microarchitectures, there already exist tools to automatically obtain those values; however, the extension of performance modeling tools like the Open Source Architecture Code Analyzer (OSACA) to support further architectures like RISC-V emphasizes the need for a multi-platform microbenchmarking tool. We present WINIC (What I Need Is Cycles), an automatic cross-platform microbenchmarking tool. Unlike previous tools, it can obtain throughput and operand-based latency metrics for most supported instructions of any 64-bit x86, AArch64, and RISC-V microarchitecture without the need for root permissions or extended system control levels. We cover the methodology of microbenchmarking and the approach of WINIC to collect cycle-accurate performance data for assembly instructions. We show that WINIC can be used to automatically obtain a vast number of accurate results and demonstrate for four target microarchitectures that our results match between 71% and 99% of the performance data of instructions obtained by similar tools and documentation. Furthermore, we investigate how operand-based metrics measured by WINIC can be used to improve the in-core performance predictions on selected micro-kernels, exemplarily for the Neoverse V2 microarchitecture.

0.35

When LLM Decompilers Recompile More and Preserve Less

cs.CR cs.AI ·2026-09-04 ·Chang Liu, Edward Raff, Kristopher Micinski
similarity 0.69
Abstract

Decompilation recovers high-level source from compiled machine code and serves as a foundation for security tasks such as vulnerability detection and malware analysis. Traditional decompilers like Ghidra and Hex-Rays expose whatever they cannot resolve as visible placeholders and often emit pseudocode that will not compile or execute; LLM-based decompilers produce clean, idiomatic C and are now judged almost entirely by recompilability and re-executability: whether the output builds and passes its shipped input/output tests. We show that these metrics can reward the wrong path: a function may recompile and pass every shipped test yet diverge on other legitimate inputs, and a disclosed vulnerability may disappear from the recompiled code with no visible trace of the crash. Neither failure is caught by existing suites. To address this gap, we propose Decompile-Diverge, a behavioral comparison oracle not relying on fixed or hand-crafted tests: for each function it synthesizes a driver, grows a fuzzing corpus from the reference, and reruns the decompiled code on the same inputs to detect changes in the function's behavior. Across eight systems in nine configurations on established LLM decompilation corpora, candidates that pass every shipped test still diverge from the original on our input corpus: 4.9% overall, and as many as 13% for a single system. On 300 real GitHub library functions and 287 CVE-grounded functions, recompilability and behavioral agreement can come apart: the strongest refinement LLM lifts Ghidra's build rate from 75% to 90%, while its Matched rate falls from 74% to 62%; on disclosed vulnerabilities, up to one tenth exhibit Crash Absence in its output. Source-level analysis traces this divergence to introduced fields, types, callees, and guards that replace the visible unknowns traditional tools leave behind.

0.38

SPEC CPU2026 Under the Microscope: Programs, Phases, and Samples

cs.AR cs.PF ·2026-11-12 ·Peijun Li, Pei-Hung Lin, Tianming Cui, Caiwen Ding, Yang Katie Zhao +1 more
benchmark similarity 0.68
Abstract

SPEC CPU2026, the latest generation of the SPEC CPU benchmarks, is set to become the standard benchmark suite in computer architecture research, where new designs are often evaluated through sampled simulation with tools such as SimPoint. Existing characterizations of the suite focus on whole-run averages, which reveal neither how behavior changes within a run nor how well sampling captures it. We characterize CPU2026 on real hardware at three levels: programs, phases, and samples. At the program level, we report whole-run metrics and top-down bottleneck classes for all four suites and examine how they change with workload size, compiler version, and platform, across machines with three different instruction set architectures: x86, Arm, and RISC-V. Notably, changing only the compiler version can halve or nearly double the IPC of individual benchmarks. At the phase level, we track hardware counters every 10 ms over all 90 SPECrate inputs, identify recurring patterns of time-varying behavior, and divide the inputs into three groups: steady, multi-phase, and bursty. More than half of the inputs show time-varying behavior, and IPC within a single input can vary by more than an order of magnitude. At the sample level, we validate SimPoint against hardware measurements of every interval it can select. SimPoint is accurate on steady inputs, but its error reaches 43% on the others, and changing only the random seed spreads the error of one input over a range of 58 percentage points. Keeping SimPoint's clusters but choosing each representative by measured CPI brings every input within 2.6%, so the error comes from the choice of representatives, not from the clustering. We conclude with practical recommendations on sampling strategy for CPU2026, which give researchers a deeper understanding of the suite and show where additional simulation effort is needed to keep sampled results accurate.

0.18

Behavioral Containment: Source-to-Binary Function Matching for 1-Day Vulnerability Detection

cs.SE ·2026-11-19 ·Yifei Yang, Xuejiao Luo, Penghui Li
code/data similarity 0.68
Abstract

Vulnerabilities in reused open-source code often propagate into deployed binaries. Detecting them requires source-to-binary function matching, which locates the affected source function in a binary. Applying prior binary-to-binary approaches to this task would require compiling the source code with the same build configuration as the target binary, which is usually unavailable. Decompiling the binary does not make prior source-to-source approaches work either, because decompiled code differs substantially from the original source code. We observe that compilation transforms the code of a function but largely preserves its behavior. A binary function that implements a source function therefore contains its behavior, a property we call behavioral containment. Building on this, we present SBIND, which decompiles the binary and checks behavioral containment instead of comparing code. SBIND represents each function by behavioral paths, sequences of normalized operations from its entry to its exits, and aligns source paths with binary paths in order. To detect 1-day vulnerabilities, SBIND compares the binary with the vulnerable and the patched source only on the paths through the changed lines. On stripped binaries built with two compilers, two optimization levels, and four architectures, SBIND ranks the correct function first in 77.87% of 32,008 searches, against 45.69% for the state-of-the-art approach SBridge. On 26 1-day CVEs, SBIND reaches an F1 score of 0.70 on stripped binaries, against 0.51 for SBridge. Our replication of SBridge also revealed several issues in its implementation and evaluation, and its authors have acknowledged them.

0.17

Discovering Performance Archetypes: Critical-Path-Aware Pattern Analysis and Regression Detection

cs.PF ·2026-09-15 ·Kaveh Shahedi, Heng Li, Maxime Lamothe, Foutse Khomh
top venue similarity 0.68
Abstract

Software performance analysis and prediction requires integrating multiple signals, as code structure alone cannot capture runtime behavior shaped by execution frequency, resource contention, and I/O patterns. We present a critical-path-aware performance analysis methodology that automatically discovers recurring performance patterns by synthesizing static code features, dynamic execution traces, and kernel-level resource data. In a preliminary study across six real-world C/C++ applications (SQLite, OpenSSL, Zstandard, FFmpeg, cURL, and jq), we first empirically confirm that static complexity metrics explain only 10.4% of the variance ($ρ^2$) in critical path execution time, quantifying a gap that, while theoretically expected, had not been measured systematically across applications. Motivated by this finding, we analyze nearly 80,000 critical execution paths and address two research questions. First, we discover 13 distinct performance archetypes: recurring behavioral patterns that appear consistently across different applications, independent of their domain or implementation. Five of these patterns are near-universal and appear in at least five of the six applications studied. Notably, three of these archetypes are present in all six applications, and together, these common patterns account for 56.4% of all observed paths. Each archetype maps to specific resource profiles and optimization strategies that transfer across domains. Second, we leverage these archetypes within a multi-signal regression detection framework that triangulates path structure, resource consumption, and archetype deviations, achieving an F1-score of 0.867 and a 60.4% improvement over resource-only methods.

0.37

FreeSBee: A Constant-Time Compilation Pipeline

cs.CR ·2026-10-14 ·Julian Ganz, David Knothe, David Oswald, Sebastian Reiter, Thorben Schneider +1 more
similarity 0.68
Abstract · 7 pages; removed incorrect conference reference

Constant-time execution is an important security property, especially for IoT devices, that prevents the leakage of sensitive information over timing side-channels. However, achieving constant-time execution is very hard when going down to the cycle level, at least on realistic devices. A lot of research is concerned with producing tools that either determine whether code is constant-time or that ensure it is compiled into a constant-time binary, with various degrees of accuracy. In this work, we present and evaluate a full workflow for hardening against timing side-channels during compilation. It starts with taint analysis to determine the code locations that depend on sensitive information. These are then the object of automated control-flow transformations inside the compiler to seal possible timing side-channels. We evaluate our workflow in detail on a realistic application running on a RISC-V core using both static timing analysis and statistical methods. Even though we do not reach cycle-accurate constant-time execution, our statistical results suggest that our hardened binary is safe against timing attacks.

0.17