CS Paper Radar

SoK: A Large-Scale Empirical Study of Emulation-Based Dynamic Analysis Research for ARM Cortex-M Firmware (Extended Version)

Hongyuan Li, Ke Wang, Wei Zhou, Le Guan

cs.CR ·submitted 2026-09-30 03:20 UTC · v1
publishedsurvey

Abstract

Microcontroller (MCU)-based devices are increasingly pervasive, making efficient, scalable firmware security analysis critical. Recent firmware re-hosting work enables automated vulnerability assessment, yet two gaps remain. First, existing tools are evaluated on limited, heavily overlapping datasets, undermining the generalizability of reported results. Second, prior research advances the state of the art along isolated dimensions, such as emulation, fuzzing, or bug diagnosis, without a holistic understanding of how these components interact and complement one another in dynamic analysis workflows. Building on recently released large-scale MCU firmware datasets from OTACAP and FirmLine, we present an empirical study of 24 emulation-based firmware analysis tools published in top conferences and journals. We position these tools within a unified automated analysis pipeline: emulation configuration reconnaissance, emulation, bug finding, and diagnosis. At each stage, we evaluate whether a tool's output provides sufficient information to enable the subsequent stage, using a deduplicated and validated subset of 4,571 ARM-based firmware samples. Only 1,580 samples (34.5%) can be successfully fuzzed, even counting a sample successful if at least one existing tool can fuzz it. Among these, fuzzing results are generally poor, with average code coverage of only 10% and many false crashes/hangs. Through a systematic analysis of failed fuzzing attempts and false-positive cases, we expose fundamental challenges and methodological limitations in current approaches. These findings highlight critical gaps in the state of the art and provide actionable insights to guide future research in emulation-based firmware analysis.

Comment
Extended Version of NDSS 2027 Accept Paper

Closest papers

Nearest neighbours by meaning across the whole index.

BullsEye: Directed Firmware Fuzzing

cs.CR ·2026-08-18 ·Lorenzo Ralli, Emilio Coppa
similarity 0.71
Abstract

The widespread adoption of Internet of Things (IoT) devices has expanded the digital attack surface, making firmware analysis critical for modern software security. A key security concern stems from the frequent reuse of third-party software components, a practice that often introduces known vulnerabilities into firmware images. Whether a given image actually exposes such a flaw is an open question, and public proof-of concept exploits make answering it urgent. Directed Greybox Fuzzing (DGF), a technique that enables targeted exploration of specific binary locations, offers a promising solution for detecting such vulnerabilities. However, DGF has reached firmware only at function granularity, too coarse to aim at the vulnerable block itself. This article presents BULLSEYE, the first DGF framework to schedule closed-source Linux-based firmware fuzzing by basic-block-level distance to user-specified targets. Our methodology combines static and dynamic analysis to enable DGF in the constrained firmware domain, focusing on vulnerabilities in reused third-party components. We introduce novel DGF heuristics that address limitations of traditional approaches. We compare BULLSEYE against four greybox-fuzzing baselines sharing its execution back-end, including reimplementations of AFLGO and WINDRANGER, and against GREENHOUSE, a state-of-the-art firmware re-hosting framework. On 40 vulnerability sites across 32 firmware images, BULLSEYE reproduces every target within budget, against 35 for the strongest of the four baselines, and reduces Time-to-Exposure by a geometric mean of 9.5x to 72.5x over them; against GREENHOUSE, on the 18 targets its pipeline supports, BULLSEYE is faster by a geometric mean of 9.8x.

0.15

IoTPwnBench: A Synthetic Benchmark for IoT Firmware Security Analysis

cs.CR cs.SE ·2026-11-11 ·Luca Ferrari, Marco Alecci, Luca Verderame, Jacques Klein, Francesco Pagano
benchmarkcode/data similarity 0.69
Abstract

Building and evaluating security tools for IoT firmware requires targets whose weaknesses are known in ad- vance and can be reproduced in a running device. Two problems remain: (1) firmware collected from real-world products provides realistic software, but hardware dependencies and undocumented defects make it difficult to establish what a security tool should be expected to find; and (2) existing benchmarks typically collect sets of vulnerable or misconfigured firmware images without providing purpose-built vulnerable services with explicit weaknesses that a security analysis tool can target. We present IoTPwnBench, a synthetic, modular benchmark that brings these requirements together. IoTPwnBench contains 716 vulnerable firmware images covering 86 weakness scenarios and 28 CWE classes across 11 device profiles, including industrial controllers, health monitors, and household devices. Each vulnerable image runs a plausible device application with simulated sensors and actuators that generate live readings over its native interfaces while exposing one target weakness mapped to a primary CWE. A documented exploitation payload and observable signal establish the expected behavior independently of the evaluated tool. Every vulnerable image is paired with a corrected twin that preserves the device and service while removing that weakness. Researchers can therefore compare the same target before and after correction to assess detection and false positives for the target weakness. The benchmark produces kernel and filesystem binaries for static inspection and dynamic testing under QEMU. Device profiles and weakness modules remain separate, allowing the same scenario to be reused across device contexts. Vulnerable targets were validated manually, and the open source build workflow regenerates the firmware bit-for-bit.

0.33

POZZER: A Power Side Channel-guided Fuzzer for Black-Box Embedded Systems

cs.CR ·2026-09-20 ·Pouya Narimani, Kseniia Rogova, Addison Crump, Martin Mohl, Meng Wang +3 more
similarity 0.68
Abstract

Firmware fuzzing is an effective technique for discovering vulnerabilities in embedded systems. However, existing coverage-guided firmware fuzzers typically obtain feedback through firmware instrumentation, hardware debug interfaces, or firmware rehosting, which requires access to the firmware source code or binary image. Such requirements are often infeasible for off-the-shelf embedded devices, where firmware binaries are inaccessible, unrehostable, immodifiable, or undebuggable, necessitating fuzzing under black-box conditions. In this paper, we present POZZER, a power side-channel-guided fuzzer for black-box embedded systems. POZZER uses power traces as feedback to identify previously unseen behavior via an incrementally constructed graph-based representation of observed executions, guiding the fuzzer toward unexplored execution paths. Its non-profiling design requires neither prior firmware knowledge nor a clone device, extracting meaningful feedback from a single power trace per execution while remaining robust to measurement noise. We evaluate POZZER on 15 firmware targets across two platforms and two real-world commercial embedded devices. Across the resulting target-platform combinations, POZZER outperforms a blind fuzzer under the same time budget in 26 out of 30 target-platform combinations. Furthermore, POZZER discovers two previously unknown vulnerabilities in one of the commercial devices, both confirmed by the vendor, demonstrating its potential for identifying vulnerabilities in black-box embedded systems.

0.18

EmbedGym: Letting Coding Agents Develop and Debug Embedded Firmware Without Physical Hardware

cs.SE cs.AI ·2026-11-16 ·Minghui Zhao, Yiming Li, Sophie Zhu, Maxwell Khor, Lauren Ouyang +3 more
similarity 0.63
Abstract

Modern coding agents perform well on general software benchmarks because they can run code and unit tests in a closed loop. Embedded firmware development, however, breaks this loop: a program that compiles can still fail on real hardware due to pin polarity, bus protocol, timing, or networking bugs that an agent at a text terminal cannot observe. Existing embedded coding agents either skip hardware execution, drive GPIO-only simulators with multi-second delays, or test on physical boards in a static environment. In this paper, we present EmbedGym, an interactive virtual hardware environment that lets coding agents develop and debug embedded firmware without physical hardware. EmbedGym uses instruction-level microcontroller unit (MCU) emulators to execute the same binary that would be flashed onto a physical board, connects them to 41 peripheral models, an analog circuit simulator, and virtual WiFi/Bluetooth Low Energy (BLE) endpoints, and exposes an interface for agents to change sensor inputs, press buttons, and inspect hardware states. On top of it, we build EmbedGym-Bench, which consists of 598 tasks across 6 MCU-framework pairs graded by hidden timed test vectors, using a five-stage multi-agent workflow that synthesizes peripheral models and tasks from datasheets and driver libraries. We expose EmbedGym to a coding agent at three levels and evaluate eight models over more than 14K trajectories. Even when compilation succeeds 99.7% of the time, 19.9% of compiled firmwares fail at runtime (up to 73% on individual targets). In contrast, interactive hardware testing in EmbedGym increases pass rates across models by +4.9% to +14.7% (up to +31% on individual targets). Our results also show that an agent with EmbedGym outperforms prior embedded agent frameworks, and lightweight models with EmbedGym match frontier models without it at 30-70x lower cost.

0.28

DeepDiff: Neuro-Dynamic Regression-Freedom Checking for Security Patch Validation

cs.CR cs.SE ·2026-10-28 ·Laboni Sarker, Shuvendu K Lahiri
datasetcode/data similarity 0.62
Abstract · 22 pages, 4 figures

Automatically validating AI-generated security patches remains a significant challenge. Eliminating a reported crash and passing existing test suites do not provide sufficient confidence that a patch correctly addresses the underlying vulnerability without introducing regressions. We present DeepDiff, a technique for regression-freedom checking that differentially evaluates candidate patches against the original vulnerable program using non-crashing inputs that exercise the affected code paths. The expectation of regression-freedom is that a correct patch should eliminate the vulnerability triggering crash while preserving the intended behavior on non-crashing inputs. To support this evaluation, we introduce a dataset of correct and incorrect patch candidates for open-source memory-safety vulnerabilities from CyberGym. We explore different approaches for non-triggering test input generation to reach the patched code, a purely neural regression-freedom checker, and a hybrid approach that combines dynamic analysis with neural techniques. Our experimental results demonstrate the effectiveness of regression-freedom checking in validating security patches while outlining the challenges that remain.

0.28

SoK: You Find What You Seek: Rethinking Oracles, Guidance, and Input Generation in Hardware Fuzzing

cs.CR cs.AR ·2026-09-23 ·G Abarajithan, Zhenghua Ma, Cristian Tirelli, Andres Meza, Francesco Restuccia +2 more
survey similarity 0.61
Abstract · 13 pages

Hardware fuzzing is an active area in security verification research, yet its industrial adoption remains in its early stages. This SoK examines which lessons from software fuzzing carry over to hardware and where unique approaches are needed. By analyzing 52 fuzzers across RTL/IP, CPU, NoC, and SoC designs, we introduce an analytical framework that frames verification as a bounded search. This search is defined by its objective, oracle, guidance, input generation, target abstraction, and budget. Consequently, a campaign only uncovers failures it can effectively reach, recognize, and prioritize before exhausting its resources. We distinguish two roles for hardware fuzzing: (1) augmenting constrained-random verification (CRV) via feedback-guided coverage and (2) directed adversarial testing based on threat models and security specifications. Through our framework, we identify what each campaign can observe and generate, providing a basis for assessing the evidence behind reported results. Our analysis suggests that mainstream adoption of hardware fuzzing will require reusable interfaces, target-specific verification assets, reproducible evaluations, and transparent reporting of cost and user effort.

0.09

VulDec: Vulnerability Detection in Binaries via Neural Decompilation

cs.CR cs.LGcs.SE ·2026-10-24 ·Francesco Crescenzo Grasso, Pietro Liguori, Roberto Natella, Domenico Cotroneo
code/data similarity 0.61
Abstract · Submitted to Automated Software Engineering

Automated vulnerability detection in binary executables is challenging due to the lack of source code and the limited semantic information available at the assembly level. Despite significant progress in machine-learning-based code analysis, the field still lacks methodologies and large, high-quality datasets that provide vulnerable code jointly at assembly- and source-level with precise vulnerability labels. To address this gap, we introduce VulDec, consisting of (i) a new large-scale, carefully curated dataset aligning real-world Assembly-C function pairs with detailed CWE annotations across multiple architectures, and (ii) the first end-to-end framework that leverages neural decompilation for binary vulnerability detection. Tuned LLM-based neural decompilers trained on our dataset achieve strong source-level recovery across architectures (Edit Similarity of approximately 56%, BLEU-4 of approximately 43%), while our tuned LLM-based detectors reach a macro F1-score of 0.61 on an in-distribution realistic test set (3% vulnerability rate). Our findings provide evidence that neural decompilation offers a powerful alternative to conventional low-level binary representations, facilitating robust binary vulnerability detection.

0.35

ReproBench: Benchmarking LLM Agents on Reproducing Vulnerability From Scratch

cs.CR ·2026-09-28 ·Liang He, Sheng Wu, Haomiao Hao, Hongduo Zhao, Jia Yan +1 more
benchmark similarity 0.61
Abstract · 15 pages (9 pages main text + 6 pages appendix), 5 figures, 11 tables. Submitted to AAAI 2027

Large language model (LLM) agents are increasingly evaluated on cybersecurity tasks such as vulnerability reproduction, exploitation, and patching. However, existing cybersecurity benchmarks predominantly operate under a post-environment evaluation paradigm, i.e., handing the agent source code, a container, or an executable binary. This setup bypasses the critical environment reconstruction step, leaving a fundamental question for real-world vulnerability analysis: can an agent autonomously reconstruct the required execution environment and reproduce a vulnerability entirely from scratch? To address this gap, we present ReproBench, an evidence-grounded benchmark designed to evaluate agent capabilities in end-to-end vulnerability reproduction starting from solely a CVE identifier. ReproBench decomposes the full reproduction workflow into six distinct phases, and assesses performance on each phase independently using verifiable experimental artifacts: downloaded firmware images, unpacked binaries, granular analysis logs, and validated crash samples, among others. We instantiate ReproBench with 30 real-world IoT firmware vulnerabilities, which serve as ideal test cases for our from-scratch evaluation setting. Our evaluation demonstrates that 45.3% of test runs resort to vulnerability simulation - a prevalent remediation workaround adopted across all evaluated LLM agents - while only 5.3% of CVE-model pairs achieve successful reproduction of real-world vulnerabilities. Despite the low overall success rate, these non-trivial successful cases confirm that state-of-the-art LLM agents already possess the capacity for fully autonomous end-to-end vulnerability reproduction. Concurrently, our in-depth analysis of failed cases identifies core bottlenecks impeding LLM agents throughout the reproduction pipeline, offering actionable insights for subsequent research.

0.41