CS Paper Radar

IoTPwnBench: A Synthetic Benchmark for IoT Firmware Security Analysis

Luca Ferrari, Marco Alecci, Luca Verderame, Jacques Klein, Francesco Pagano

cs.CR cs.SE ·submitted 2026-11-11 10:55 UTC · v1
benchmarkcode/data

Abstract

Building and evaluating security tools for IoT firmware requires targets whose weaknesses are known in ad- vance and can be reproduced in a running device. Two problems remain: (1) firmware collected from real-world products provides realistic software, but hardware dependencies and undocumented defects make it difficult to establish what a security tool should be expected to find; and (2) existing benchmarks typically collect sets of vulnerable or misconfigured firmware images without providing purpose-built vulnerable services with explicit weaknesses that a security analysis tool can target. We present IoTPwnBench, a synthetic, modular benchmark that brings these requirements together. IoTPwnBench contains 716 vulnerable firmware images covering 86 weakness scenarios and 28 CWE classes across 11 device profiles, including industrial controllers, health monitors, and household devices. Each vulnerable image runs a plausible device application with simulated sensors and actuators that generate live readings over its native interfaces while exposing one target weakness mapped to a primary CWE. A documented exploitation payload and observable signal establish the expected behavior independently of the evaluated tool. Every vulnerable image is paired with a corrected twin that preserves the device and service while removing that weakness. Researchers can therefore compare the same target before and after correction to assess detection and false positives for the target weakness. The benchmark produces kernel and filesystem binaries for static inspection and dynamic testing under QEMU. Device profiles and weakness modules remain separate, allowing the same scenario to be reused across device contexts. Vulnerable targets were validated manually, and the open source build workflow regenerates the firmware bit-for-bit.

Closest papers

Nearest neighbours by meaning across the whole index.

BullsEye: Directed Firmware Fuzzing

cs.CR ·2026-08-18 ·Lorenzo Ralli, Emilio Coppa
similarity 0.73
Abstract

The widespread adoption of Internet of Things (IoT) devices has expanded the digital attack surface, making firmware analysis critical for modern software security. A key security concern stems from the frequent reuse of third-party software components, a practice that often introduces known vulnerabilities into firmware images. Whether a given image actually exposes such a flaw is an open question, and public proof-of concept exploits make answering it urgent. Directed Greybox Fuzzing (DGF), a technique that enables targeted exploration of specific binary locations, offers a promising solution for detecting such vulnerabilities. However, DGF has reached firmware only at function granularity, too coarse to aim at the vulnerable block itself. This article presents BULLSEYE, the first DGF framework to schedule closed-source Linux-based firmware fuzzing by basic-block-level distance to user-specified targets. Our methodology combines static and dynamic analysis to enable DGF in the constrained firmware domain, focusing on vulnerabilities in reused third-party components. We introduce novel DGF heuristics that address limitations of traditional approaches. We compare BULLSEYE against four greybox-fuzzing baselines sharing its execution back-end, including reimplementations of AFLGO and WINDRANGER, and against GREENHOUSE, a state-of-the-art firmware re-hosting framework. On 40 vulnerability sites across 32 firmware images, BULLSEYE reproduces every target within budget, against 35 for the strongest of the four baselines, and reduces Time-to-Exposure by a geometric mean of 9.5x to 72.5x over them; against GREENHOUSE, on the 18 targets its pipeline supports, BULLSEYE is faster by a geometric mean of 9.8x.

0.15

SoK: A Large-Scale Empirical Study of Emulation-Based Dynamic Analysis Research for ARM Cortex-M Firmware (Extended Version)

cs.CR ·2026-09-30 ·Hongyuan Li, Ke Wang, Wei Zhou, Le Guan
publishedsurvey similarity 0.69
Abstract · Extended Version of NDSS 2027 Accept Paper

Microcontroller (MCU)-based devices are increasingly pervasive, making efficient, scalable firmware security analysis critical. Recent firmware re-hosting work enables automated vulnerability assessment, yet two gaps remain. First, existing tools are evaluated on limited, heavily overlapping datasets, undermining the generalizability of reported results. Second, prior research advances the state of the art along isolated dimensions, such as emulation, fuzzing, or bug diagnosis, without a holistic understanding of how these components interact and complement one another in dynamic analysis workflows. Building on recently released large-scale MCU firmware datasets from OTACAP and FirmLine, we present an empirical study of 24 emulation-based firmware analysis tools published in top conferences and journals. We position these tools within a unified automated analysis pipeline: emulation configuration reconnaissance, emulation, bug finding, and diagnosis. At each stage, we evaluate whether a tool's output provides sufficient information to enable the subsequent stage, using a deduplicated and validated subset of 4,571 ARM-based firmware samples. Only 1,580 samples (34.5%) can be successfully fuzzed, even counting a sample successful if at least one existing tool can fuzz it. Among these, fuzzing results are generally poor, with average code coverage of only 10% and many false crashes/hangs. Through a systematic analysis of failed fuzzing attempts and false-positive cases, we expose fundamental challenges and methodological limitations in current approaches. These findings highlight critical gaps in the state of the art and provide actionable insights to guide future research in emulation-based firmware analysis.

0.22

Cross-Corpus Evaluation of Generalizable Vulnerability Detection in IoT Firmware

cs.CR cs.LG ·2026-08-11 ·Sadib Hassan Rumman, Md. Shariful Islam, Md Rayhanur Rahman
benchmark similarity 0.67
Abstract · 7 pages, 1 Figure, 6 Tables

IoT firmware vulnerability detection is constrained by ecosystem heterogeneity, resource-limited platforms, and benchmark quality limitations. Existing datasets are often synthetic or general-purpose and lack human-verified, contamination-screened annotations, leaving cross-corpus generalization across training sources, architectures, and curriculum design underexplored. In this study, we have introduced IoTVulBench, a human-verified benchmark for cross-corpus firmware vulnerability detection. IoTVulBench was built from GitHub repositories, validated by three expert reviewers, and evaluated on a contamination-screened held-out target across five architectures, two tuning methods, and three curriculum strategies, with ensemble, distillation, and robustness analyses. Models trained on IoTVulBench reached the highest Matthews Correlation Coefficient (MCC) among undersampling-matched single-source datasets, at 0.58 versus 0.44 for PrimeVul and 0.39 for D2A. Staged curriculum learning raised MCC to 0.69, and a diversity-optimized ensemble reached 0.73. This gain represents a 0.42 MCC improvement over the strongest reference comparator, a static analyzer with an MCC of 0.31, and a 0.29 MCC improvement over the strongest single-source dataset, PrimeVul. At a 0.5% false-positive rate, the model missed only 21% of vulnerabilities versus 71% for the comparator. The model also retained 86% of its performance under identifier renaming, with strong calibration. These results indicate that domain-matched training data and curriculum design, rather than model scale alone, drive generalization in firmware vulnerability detection, and yield both a benchmark and deployment-ready configurations for IoT security.

0.07

Beyond Smart Homes: Characterizing IoT Vulnerabilities Across Sectors, Vendors, and Products

cs.CR ·2026-10-30 ·Tina Rezaei, Suzan Bayhan, Andrea Continella, Jeroen van der Ham-de Vos, Roland van Rijswijk-Deij
similarity 0.65
Abstract

The Internet of Things (IoT) has expanded beyond consumer smart homes to sectors, such as industrial systems, healthcare, and smart cities, broadening the IoT attack surface. Prior empirical work has focused on studying aggregate IoT vulnerability characteristics or specific protocols, so patching is directed by aggregate security guidance. However, if dominant weaknesses differ substantially across sectors or products, such uniform guidance misdirects the remediation effort. To address this gap, we present a fine-grained, data-driven analysis of IoT vulnerabilities using a dataset of more than 15,000 IoT specific CVEs. We design a pipeline to enrich noisy CPE (Common Platform Enumeration) metadata, normalizing vendors and products, then assigning product type and sector labels using lexical matching, fuzzy clustering, and LLM-assisted classification. We find meaningful differences in the weakness profile of IoT sectors. For example, healthcare is dominated by authentication and credential weaknesses, while automotive faces memory issues. More importantly, we find that weakness profiles are more similar across different product types from the same vendor than across the same product type built by different vendors. This suggests vendor-level factors such as shared code and development practices dominate weakness inheritance, making a vendor's disclosure history informative about its other products. Our findings suggest weakness classes should be used to prioritize remediation efforts.

0.35

ReproBench: Benchmarking LLM Agents on Reproducing Vulnerability From Scratch

cs.CR ·2026-09-28 ·Liang He, Sheng Wu, Haomiao Hao, Hongduo Zhao, Jia Yan +1 more
benchmark similarity 0.64
Abstract · 15 pages (9 pages main text + 6 pages appendix), 5 figures, 11 tables. Submitted to AAAI 2027

Large language model (LLM) agents are increasingly evaluated on cybersecurity tasks such as vulnerability reproduction, exploitation, and patching. However, existing cybersecurity benchmarks predominantly operate under a post-environment evaluation paradigm, i.e., handing the agent source code, a container, or an executable binary. This setup bypasses the critical environment reconstruction step, leaving a fundamental question for real-world vulnerability analysis: can an agent autonomously reconstruct the required execution environment and reproduce a vulnerability entirely from scratch? To address this gap, we present ReproBench, an evidence-grounded benchmark designed to evaluate agent capabilities in end-to-end vulnerability reproduction starting from solely a CVE identifier. ReproBench decomposes the full reproduction workflow into six distinct phases, and assesses performance on each phase independently using verifiable experimental artifacts: downloaded firmware images, unpacked binaries, granular analysis logs, and validated crash samples, among others. We instantiate ReproBench with 30 real-world IoT firmware vulnerabilities, which serve as ideal test cases for our from-scratch evaluation setting. Our evaluation demonstrates that 45.3% of test runs resort to vulnerability simulation - a prevalent remediation workaround adopted across all evaluated LLM agents - while only 5.3% of CVE-model pairs achieve successful reproduction of real-world vulnerabilities. Despite the low overall success rate, these non-trivial successful cases confirm that state-of-the-art LLM agents already possess the capacity for fully autonomous end-to-end vulnerability reproduction. Concurrently, our in-depth analysis of failed cases identifies core bottlenecks impeding LLM agents throughout the reproduction pipeline, offering actionable insights for subsequent research.

0.41

CESBench: Benchmarking Large Language Models on Cryptographic Engineering Security for IoT Devices

cs.CR cs.AI ·2026-09-18 ·Wenquan Zhou, An Wang, Jing Liang, Peien Feng, Jingqi Zhang +2 more
benchmark similarity 0.63
Abstract

For Internet of Things (IoT) devices, a secure algorithm alone is not enough: an attacker with physical access can attack the implementation directly, and its flaws are hard to fix once deployed. Large language models (LLMs) are now used to build and analyze such implementations. LLM benchmarks exist for cryptography and general cybersecurity, but none covers cryptographic engineering. In this paper, we present CESBench, 380 expert-written items across six sub-domains of cryptographic engineering security for IoT devices: side-channel, fault injection, implementation, countermeasures, evaluation, and integration. Four task types target different competences: 209 multiple-choice items test recall, 67 judgment items require a security verdict and its justification, 63 scenario items require an engineering diagnosis, and 41 code tasks are graded by 572 test cases. To validate the benchmark, 11 open-weight and proprietary LLMs answer every item. Multiple-choice and code responses are scored automatically, and judgment and scenario responses by an LLM judge, whose scores are checked against a second judge from another model family and human re-scoring. Composite scores range from 54.4% to 83.6%. The top score on each task type is 98.6% for multiple choice, 95.1% for code, and 88.4% for scenario diagnosis, but only 58.8% for judgment. Across models, 88.5% of verdicts are correct, yet their justifications earn only 53.4% of the rubric marks. Multiple choice is near its ceiling for the strongest models and most code tasks are solved, whereas justifying a security verdict remains the weakest competence. The benchmark, prompts, and per-item results are public.

0.20

From Source Code to Network Profile: Automated and Traceable MUD Profile Generation for IoT Devices

cs.CR cs.SE ·2026-09-25 ·Alessandro Lotto, Abdulla R. A. Almenhali, Savio Sciancalepore, Alessandro Brighente, Mauro Conti
similarity 0.61
Abstract

The Manufacturer Usage Description (MUD) standard allows IoT manufacturers to define expected network behaviors in a MUD file. This file can be translated into enforceable access-control policies, restricting compromised devices to operate solely through manufacturer-defined communication patterns. However, practical adoption of MUD depends on profiles that are accurate, complete, and maintainable. Existing approaches use traffic-based automation but require device deployment and prolonged monitoring, capturing only behavior exercised during observation. Rare, failure-triggered, or configuration-dependent communications may remain absent, producing incomplete policies that disrupt legitimate operation and offer limited insight into the software components responsible for each rule. We present AutoMUD, a source-code-driven tool that generates traceable MUD profiles for IoT devices from their firmware and software source code. AutoMUD combines static and syntactic extraction, retrieval-grounded language-model reasoning, and deterministic validation and compilation to recover the communication behavior characterizing an IoT device and translate eligible endpoints into policy rules. By analyzing code-level evidence, AutoMUD exposes rarely exercised and conditional communication paths, links every generated rule to its source-level provenance, and preserves excluded findings with explicit reasons for review. Our evaluation on a Linux-based repository demonstrates that AutoMUD recovers complete communication behavior, consolidates validated behavior into semantic endpoint groups, and generates structurally valid MUD profiles. Through a controlled semantic fault-injection campaign, we demonstrate that AutoMUD enables analysts to detect, localize, explain, and correct propagated errors, recovering policies semantically identical to their clean counterparts.

0.20

HoliBench: A Cross-Platform Benchmarking and Deployment Toolkit for Foundation Models in CPS-IoT Applications

cs.DC cs.LGcs.PF ·2026-09-11 ·Inesh Chakrabarti, Zejun Xiong, Pragya Sharma, Mani Srivastava
code/data similarity 0.61
Abstract · 16 pages, 9 figures, 2 tables. Code: https://github.com/beesfleas/HoliBench

Foundation models, including large language models, vision-language models, and time-series foundation models, are increasingly deployed on embedded and edge platforms for CPS and IoT applications, where energy, latency, and memory are as critical as task accuracy. Existing benchmarking tools evaluate model capability in isolation, reporting accuracy assuming sufficient compute, while hardware profiling tools remain platform-specific and mutually incompatible. As a result, users lack a unified workflow for making deployment decisions across heterogeneous devices. We present HoliBench, a modular benchmarking and deployment toolkit that jointly characterizes accuracy, latency, and energy across platforms from single-board computers to GPU servers. Its platform abstraction layer calibrates cross-device measurement, and the toolkit supports multiple model modalities, inference engines, concurrencies, and existing evaluation harnesses. An interactive interface exposes constraint-aware configuration selection over a design space that is profiled once and reused across studies. Using HoliBench, we characterize 20 models across 7 device types, 3 quantization levels, 8 inference backends, and over 30 tasks, surfacing tradeoffs that existing tools miss: quantization reduces latency only on hardware with low-precision support, accuracy gains show diminishing returns relative to energy, and for autoregressive workloads, average inference power is approximately constant across output lengths. We further find that single-model profiles compose under sequential co-resident execution. In a multi-model CPS deployment, standalone profiles predict combined-pipeline latency and power within 1.2% and 2.5%, enabling deployment exploration without exhaustively profiling every pipeline configuration. We release HoliBench as open-source infrastructure for deployment-aware evaluation of foundation models.

0.36