CS Paper Radar

MallocSan: A Memory Safety Tool for Native Closed-Source Applications

Adel Belkhiri, Michel Dagenais, Maxime Lamothe

cs.SE ·submitted 2026-09-29 20:24 UTC · v1

Abstract

Memory-corruption errors remain a leading cause of high-impact vulnerabilities in C and C++ software. Deterministic detection, however, remains difficult to deploy: compiler-based sanitizers require source code and control of the build pipeline, hardware-assisted schemes depend on specific platforms, and dynamic binary translation can impose order-of-magnitude slowdowns. This paper presents MallocSan, a heap sanitizer for native, potentially closed-source x86-64 Linux applications that requires no source access, recompilation, or specialized hardware. MallocSan interposes on memory allocation through LD_PRELOAD and embeds an object identifier in the unused high bits of each protected pointer. Dereferencing the resulting noncanonical pointer faults at the offending instruction, which MallocSan decodes and patches at runtime so that subsequent executions perform per-object bounds checks entirely in userspace. Sites that cannot be patched fall back to in-handler emulation or single-stepping, while an optional profile-guided pass rewrites frequently executed residual sites offline. MallocSan also extends identity-based checking to vector gather/scatter instructions and supports policy-scoped coverage. Across seven SPEC CPU 2017 benchmarks, MallocSan outperforms Valgrind Memcheck on five, with a geometric-mean execution-time factor of 4.82x relative to native execution, compared with 18.55x for Memcheck, while preserving substantial parallel scaling on both evaluated multithreaded workloads: 644.nab_s and pigz. On the in-scope Juliet tests, MallocSan detects all seeded violations with no false reports on the good executions. It also detects heap-buffer errors in real-world applications, including a known one-byte overread in LibTIFF's tiffcrop utility.

Comment
28 pages. Submitted to ACM TOSEM; under review

Closest papers

Nearest neighbours by meaning across the whole index.

GPU Offload in Rust: Portable, Safe, and Fast

cs.PL ·2026-08-13 ·Manuel S. Drehwald, Marcelo Domínguez, Kevin Sala, Alán Aspuru-Guzik, Johannes Doerfert
similarity 0.70
Abstract · 13 pages, 5 figures

High-performance GPU programming has traditionally forced a compromise between execution efficiency and memory safety. While Rust guarantees compile-time memory safety for host CPUs via its strict ownership model, applying these constraints to massively parallel GPU execution environments has previously mandated either vendor-locked Domain-Specific Languages (DSLs) or escaping to explicit unsafe raw pointers. This paper presents a zero-overhead, multi-vendor GPU compilation framework built natively into the Rust compiler (rustc) and LLVM backends. We leverage Rust's rich type system, ownership system, and strict aliasing guarantees (noalias) to efficiently manage and optimize data transfers through LLVM's Offload infrastructure. We expose the technical challenges of cross-vendor ABI lowering mismatches between Host and Device targets and introduce a two-pass compilation pipeline capable of safely handling both manual and compiler-generated memory movements. Evaluating our framework on RAJAPerf demonstrates that our rustc-based solution can generate competitive LLVM IR for GPU kernels, achieving a solid kernel performance against native, hand-optimized CUDA and HIP C++ baselines.

0.39

UnsafeChecker: Finding Soundness Bugs in Rust Safe Abstractions

cs.PL ·2026-09-09 ·Xizhe Yin, Yaokun Zhang, Yang Feng, Baowen Xu
benchmark similarity 0.69
Abstract

Rust guarantees memory safety without garbage collection through a strict ownership and borrowing system. However, for low-level systems programming, many widely used libraries rely on the unsafe keyword. These libraries encapsulate raw-pointer operations behind safe APIs to form safe abstractions. A single mistake in this internal unsafe code can break its safety contract, rendering the abstraction unsound and allowing safe clients to trigger undefined behavior. Detecting these potential soundness violations is challenging. Existing static analysis tools for C/C++ ignore Rust-specific safety contracts, while current Rust tools lack the deep semantic modeling required to track the contexts that raw pointers erase. To address this gap, we present UnsafeChecker, a compiler-integrated static analysis framework for detecting potential soundness violations in Rust safe abstractions. UnsafeChecker analyzes Rust MIR using a flow-sensitive abstract interpretation that maintains a shared state with three components: ownership, object validity, and layout. Each warning rule consumes the subset of facts needed for the corresponding Rust safety obligation. UnsafeChecker reports both instruction-level undefined behavior and boundary-level contract violations that may escape through safe APIs. We evaluate UnsafeChecker on a benchmark of 46 RustSec vulnerabilities, which contain 53 ground-truth bugs. UnsafeChecker outperforms several state-of-the-art tools, detecting 32 CVEs and covering 36 bugs (67.9% recall) with 51.6% alert-level precision. Furthermore, in a large-scale scan of real-world crates on crates.io, UnsafeChecker uncovered 114 previously unknown bugs across 83 crates, with 45 confirmed and 27 already fixed by maintainers.

0.09

CHERI under Fault Injection: An Emulation-Based Security Evaluation of Capability Hardware Enhanced RISC Instructions

cs.CR cs.AR ·2026-11-15 ·Eva Sophia Schreyer
similarity 0.68
Abstract · Submitted to KTH Royal Institute on Technology as a Master Thesis

Memory-safety vulnerabilities continue to dominate real-world exploit chains despite decades of research into software mitigations. CHERI is a hardware architecture that enforces spatial and temporal memory safety by augmenting pointers with hardware-checked metadata, thus preventing software-based exploitation, but its threat model explicitly excludes fault injection. This thesis investigates whether targeted fault injection can undermine CHERI's memory-safety guarantees in a pure-capability CHERI-RISC-V environment. Two fault classes are considered: bit flips corrupting capability metadata, and fault-induced check skipping on capability store instructions. We implement fault injection primitives in a QEMU-based emulation environment, which enables deterministic, repeatable experiments without specialised hardware. With them, we demonstrate four attack scenarios: buffer overflows, control-flow hijacking, data-only attacks, and use-after-free. Each succeeds with one or two targeted faults (lower bounds under the emulation-based setting). Check skipping offers an alternative route to control-flow hijacking and data-only attacks, with a minimum of one fault. These results show that CHERI's guarantees depend on the physical integrity of capability metadata, the uninterrupted execution of metadata checks, and the integrity of the revocation bitmap: a single targeted fault can circumvent spatial safety, temporal safety, or control-flow integrity. CHERI still raises the cost of exploitation, since fault injection is a prerequisite for every attack and the sentry mechanism forces an attacker to supply a valid tagged capability or spend a second fault restoring the tag. Based on these findings, we propose mitigations: ECC-protected capability memory, capability field authentication codes, redundant execution of security-critical checks, and revocation bitmap integrity verification.

0.29

Efficient Linkage-Based Compartmentalization on CHERI

cs.CR cs.ARcs.OS ·2026-09-29 ·Dapeng Gao, John Baldwin, Jessica Clarke, Nicholas C. Connolly, Brooks Davis +8 more
published similarity 0.68
Abstract

We present an efficient linkage-based model for in-process compartmentalization built on CHERI memory safety, which enables fine-grained compartmentalization of the entire UNIX user-space, scaling to 10K+ compartments on desktop systems. The model's "push-button" compartmentalization along existing library boundaries regularly hosts 500+ compartments per process for large applications such as Chromium, far exceeding the number of concurrently available protection domains supported by other mechanisms (e.g., up to 16 for Intel MPK). Custom policies can further subdivide libraries. Of the thousands of C/C++ programs tested, only the V8 JavaScript engine required source-level adaptation (<300 lines of changed code concerning garbage collection and JIT compilation). We implement the model for CHERI-extended versions of Armv8-A and RISC-V through support in the compiler toolchain and operating system. Case studies illustrate the smooth delegation of memory between compartments, compartment-aware debugging and visualization, as well as extensibility to a complex managed language runtime, demonstrating the benefits of our single-address-space model. We evaluate using multiple processors, including Arm's superscalar Morello and, notably, the first commercial CHERI-enabled RISC-V application core---Codasip's in-order dual-issue X730.

0.26

FRESCO: Complete and Scalable Temporal Safety for CHERI Application Processors

cs.CR ·2026-08-26 ·Merve Gülmez, Nils Jordan, Jialun Zhang, Hossam ElAtali, Gang Tan +2 more
similarity 0.68
Abstract

CHERI provides hardware-enforced spatial memory safety. While prior work extends it with heap temporal safety, stack use-after-return remains unaddressed. Existing defenses fall short: compiler analysis reliably catches only references that escape as function return values, while dynamic sanitizers impose overheads that preclude production deployment. We present FRESCO, built on the principle that a stack capability must not outlive the frame that created it. FRESCO "colors" the stack pointer with per-invocation provenance identifiers; every capability derived from it inherits that lifetime and is hardware-invalidated the moment the function exits, regardless of how or where it escaped. Because stack frames retire orders of magnitude more frequently than heap allocations, FRESCO manages the resulting color pressure through: 1) Color Saver, a static capability-aware escape analysis that confines coloring to functions needing it, and whose core algorithm we mechanically verify in Rocq, and 2) capability-color segmentation, which partitions memory into disjoint segments, each with an independent color namespace. Color segmentation lets stack and heap temporal safety coexist on one system, making FRESCO the first hardware/software co-design to provide complete and scalable temporal safety for CHERI application processors. We realize FRESCO on the CHERI-RISC-V QEMU full-system emulator and the out-of-order CHERI-Toooba FPGA softcore, with software support in the CHERI-enabled Clang/LLVM compiler and CheriBSD OS. FRESCO systematically prevents use-after-return, use-after-free, and double-free across the NIST Juliet Test Suite and CVEs, with only a small run-time overhead in SPEC CPU (4% g.m.), SQLite, and PostgreSQL (10-14%).

0.14

FreeSBee: A Constant-Time Compilation Pipeline

cs.CR ·2026-10-14 ·Julian Ganz, David Knothe, David Oswald, Sebastian Reiter, Thorben Schneider +1 more
similarity 0.68
Abstract · 7 pages; removed incorrect conference reference

Constant-time execution is an important security property, especially for IoT devices, that prevents the leakage of sensitive information over timing side-channels. However, achieving constant-time execution is very hard when going down to the cycle level, at least on realistic devices. A lot of research is concerned with producing tools that either determine whether code is constant-time or that ensure it is compiled into a constant-time binary, with various degrees of accuracy. In this work, we present and evaluate a full workflow for hardening against timing side-channels during compilation. It starts with taint analysis to determine the code locations that depend on sensitive information. These are then the object of automated control-flow transformations inside the compiler to seal possible timing side-channels. We evaluate our workflow in detail on a realistic application running on a RISC-V core using both static timing analysis and statistical methods. Even though we do not reach cycle-accurate constant-time execution, our statistical results suggest that our hardened binary is safe against timing attacks.

0.17

Beyond Single-Vulnerability Evaluation: Closing the Engineering Decision Gap Between C Retrofits and Native Safety

cs.PL cs.CR ·2026-08-16 ·Andrew Laramore, Joseph Spracklen, Murtuza Jadliwala
position similarity 0.67
Abstract

While decades of research have produced numerous retrofitted memory-safety protections for C, these mechanisms are almost exclusively evaluated in isolation, targeting specific vulnerability classes. This siloed evaluation paradigm leaves practitioners without a clear understanding of the cumulative performance costs, interoperability conflicts, and protection gaps that arise when layering defenses to achieve comprehensive safety. This paper presents a new evaluation paradigm that benchmarks natively memory-safe languages like Rust and Go against compounded C retrofits. Using standardized cross-language tasks, we evaluate the performance and protection tradeoffs of state-of-the-art mechanisms when deployed in combination. Our results demonstrate that layered C defenses incur compounding and workload-dependent performance penalties, can suffer from fundamental architectural incompatibilities, and fall short of the protection scope provided by native memory-safe languages. These findings expose a critical engineering decision gap where the true cost of backporting safety to C remains hidden from practitioners. We argue for a fundamental shift in memory-safety research: moving away from isolated evaluation toward holistic, comparative frameworks that inform the high-stakes choice between retrofitting legacy codebases and migrating to modern, safe languages.

0.04

PIMROP: A Feasibility Study of Return-Oriented Programming on Commercial Processing-in-Memory Architectures

cs.CR cs.AR ·2026-11-04 ·William C. Tegge, Shouzhi Fang, Alex K. Jones
code/data similarity 0.67
Abstract

Processing-in-Memory places small programmable cores on the memory device, where they execute beside the DRAM banks and beyond the control-flow defenses that protect host programs. The security of this in-memory code is an attack surface that remains largely unstudied. We present PIMROP, to our knowledge the first demonstrated code-reuse attack on a commercial Processing in-Memory ISA. We show that return-oriented programming is feasible on a real, commercially shipped PIM instruction set, the UPMEM DPU, whose DRAM Processing Units run C compiled by a dedicated LLVM toolchain. We build Galileo-for-DPU, a gadget scanner adapted to the DPU's fixed-width, 8-byte-aligned RISC encoding, and run it entirely on the UPMEM functional simulator in a reproducible Docker and VM environment. We find the runtime library that every kernel links already yields 517 non-trivial gadgets covering all five canonical ROP primitive classes, and all 13 PrIM benchmarks we scanned carry gadgets as well. We then demonstrate two verified end-to-end attacks, each reproducible on 10 of 10 runs. In the first, a stack buffer overflow in a DPU kernel overwrites the saved return address and diverts control into a routine that copies a secret into host-readable memory. In the second, we leak the same secret using a chain of reused gadgets alone, without diverting control into any such routine. We further show that two standard defenses--stack canaries and _FORTIFY_SOURCE-style bounds-check fortification--fail on the DPU in ways specific to the platform (misplaced canary, silently inactive wrapper), and we verify a bounds-checked DMA that stops the attack on 10 of 10 runs with no effect on benign requests. To enable further research on PIM security, we open-source Galileo-for-DPU and our testing harness at https://github.com/wtegge2/ROP-in-PIM.

0.28