CS Paper Radar

Cross-Corpus Evaluation of Generalizable Vulnerability Detection in IoT Firmware

Sadib Hassan Rumman, Md. Shariful Islam, Md Rayhanur Rahman

cs.CR cs.LG ·submitted 2026-08-11 23:05 UTC · v2
benchmark

Abstract

IoT firmware vulnerability detection is constrained by ecosystem heterogeneity, resource-limited platforms, and benchmark quality limitations. Existing datasets are often synthetic or general-purpose and lack human-verified, contamination-screened annotations, leaving cross-corpus generalization across training sources, architectures, and curriculum design underexplored. In this study, we have introduced IoTVulBench, a human-verified benchmark for cross-corpus firmware vulnerability detection. IoTVulBench was built from GitHub repositories, validated by three expert reviewers, and evaluated on a contamination-screened held-out target across five architectures, two tuning methods, and three curriculum strategies, with ensemble, distillation, and robustness analyses. Models trained on IoTVulBench reached the highest Matthews Correlation Coefficient (MCC) among undersampling-matched single-source datasets, at 0.58 versus 0.44 for PrimeVul and 0.39 for D2A. Staged curriculum learning raised MCC to 0.69, and a diversity-optimized ensemble reached 0.73. This gain represents a 0.42 MCC improvement over the strongest reference comparator, a static analyzer with an MCC of 0.31, and a 0.29 MCC improvement over the strongest single-source dataset, PrimeVul. At a 0.5% false-positive rate, the model missed only 21% of vulnerabilities versus 71% for the comparator. The model also retained 86% of its performance under identifier renaming, with strong calibration. These results indicate that domain-matched training data and curriculum design, rather than model scale alone, drive generalization in firmware vulnerability detection, and yield both a benchmark and deployment-ready configurations for IoT security.

Comment
7 pages, 1 Figure, 6 Tables

Closest papers

Nearest neighbours by meaning across the whole index.

IoTPwnBench: A Synthetic Benchmark for IoT Firmware Security Analysis

cs.CR cs.SE ·2026-11-11 ·Luca Ferrari, Marco Alecci, Luca Verderame, Jacques Klein, Francesco Pagano
benchmarkcode/data similarity 0.67
Abstract

Building and evaluating security tools for IoT firmware requires targets whose weaknesses are known in ad- vance and can be reproduced in a running device. Two problems remain: (1) firmware collected from real-world products provides realistic software, but hardware dependencies and undocumented defects make it difficult to establish what a security tool should be expected to find; and (2) existing benchmarks typically collect sets of vulnerable or misconfigured firmware images without providing purpose-built vulnerable services with explicit weaknesses that a security analysis tool can target. We present IoTPwnBench, a synthetic, modular benchmark that brings these requirements together. IoTPwnBench contains 716 vulnerable firmware images covering 86 weakness scenarios and 28 CWE classes across 11 device profiles, including industrial controllers, health monitors, and household devices. Each vulnerable image runs a plausible device application with simulated sensors and actuators that generate live readings over its native interfaces while exposing one target weakness mapped to a primary CWE. A documented exploitation payload and observable signal establish the expected behavior independently of the evaluated tool. Every vulnerable image is paired with a corrected twin that preserves the device and service while removing that weakness. Researchers can therefore compare the same target before and after correction to assess detection and false positives for the target weakness. The benchmark produces kernel and filesystem binaries for static inspection and dynamic testing under QEMU. Device profiles and weakness modules remain separate, allowing the same scenario to be reused across device contexts. Vulnerable targets were validated manually, and the open source build workflow regenerates the firmware bit-for-bit.

0.33

Reassessing Dataset Quality and Fine-Tuning Practices for Vulnerability Detection on PrimeVul dataset

cs.SE ·2026-11-04 ·Norbert Sándor Szolnoki, Gábor Antal
publishedbenchmark similarity 0.63
Abstract · Presented at EASE 2026 AI Models/Data track: https://conf.researchr.org/track/ease-2026/ease-2026-ai-models---data?

Recent studies report promising results from supervised fine-tuning of pre-trained code language models for automated vulnerability detection on benchmarks such as PrimeVul. In this paper, we present a large-scale empirical evaluation of both model performance and dataset quality in this setting. We systematically fine-tune three widely used encoder models (CodeBERT, GraphCodeBERT, and UniXCoder) across five imbalance mitigation strategies, extensive hyperparameter sweeps, and multiple random seeds, resulting in 770 independent training runs. Our results reveal a consistent pattern. While standard metrics such as F1 and ROC-AUC occasionally suggest improvements, security-critical and calibration-aware measures tell a different story. Matthews Correlation Coefficient remains near zero, predicted probabilities are poorly calibrated, and evaluation under strict false-positive constraints reveals high false-negative rates. Neither imbalance handling, threshold optimization, nor post-hoc calibration recovers a meaningful discriminative signal. We also conduct a data quality assessment of the PrimeVul benchmark and identify structural issues such as minimal semantic differences between vulnerable and fixed functions, and annotation noise introduced by commit-level labeling. These findings suggests that the PrimeVul dataset, in its current form, presents significant challenges for reliable AI-based vulnerability detection in software engineering with the methodologies we used. Our study highlights the need for more robust dataset design, improved evaluation protocols, and rethinking of fine-tuning practices for security-critical tasks.

0.33

Predicting CVSS and CWE from CVE Descriptions with Fine-Tuned Small Language Models

cs.LG cs.CR ·2026-10-16 ·Tara Dixit, Jessie Dong, Summer Li
similarity 0.62
Abstract · 23 pages, 5 figures, 21 tables

Until April 2026, the National Vulnerability Database labeled nearly every published Common Vulnerabilities and Exposures (CVE) entry with a Common Vulnerability Scoring System (CVSS) v3.1 vector, a severity tier, and a Common Weakness Enumeration (CWE) weakness type. Now, NIST enriches only the high priority CVEs, leaving many newly published vulnerabilities unlabelled with the classifiers that security teams use for patch triage. In CVE enrichment, we predict the eight base components of CVSS v3.1 vector, base score, severity tier, and CWE from the vulnerability's textual description alone. To prevent leakage between near-duplicate descriptions, all systems are trained only on CVEs published in 2022 and 2023. Results are evaluated on all 36,325 usable 2024 CVEs in the pinned NVD snapshot. All scores are either directly predicted, or calculated from predicted vectors using the CVSS formula. A fine-tuned 3.8B-parameter model Phi-3.5-mini, trained with LoRA in 2.5 hours on a single Apple M5 Pro chip, achieves 69.9% accuracy on severity and 70.9% on CWE. It matches the official vector exactly for 48.8% of test CVEs, outperforming TF-IDF logistic regression by 2.8 to 4.7 points on all accuracy metrics based on zero-exclusion paired bootstrap intervals. When we evaluate the base model with a zero-shot prompt, it achieves 20.8% accuracy on severity in a sample of 2,000 CVEs. In this same sample, its stated scores match its own vector precisely for only 3.4% of CVEs. However, deriving scores from predicted vectors boosts the critical recall of TF-IDF logistic regression from 50.7% to 73.3% with no additional cost. And since the errors of the fine-tuned model and TF-IDF affect different CVEs, at least one system matches the correct severity tier for 75.8% of test CVEs.

0.09

The Data Problem in Software Vulnerability Analysis: Artifacts, Quality, and Consumption

cs.SE ·2026-09-01 ·Yu Nong, Yao Du, Tianxiang Xu, Haipeng Cai
similarity 0.61
Abstract

Learning- and LLM-based software vulnerability analysis is only as trustworthy as the data it is trained and evaluated on, yet that data is rarely examined as a first-class object. We investigate the data behind vulnerability analysis through a dataset-centric taxonomy that separates what an artifact is (code, metadata, patches, tests/PoCs, reasoning, traces), how good it is (realism, label evidence, scale, diversity, leakage, availability), and what it is used for. From a systematically assembled corpus of 1522 papers covering 2016-2026 plus foundational earlier work we deep-code a tiered set of 111 anchor papers, backing every affirmative rubric-graded value with a verbatim span, and we report, per attribute, both how much it has been studied and how well datasets achieve it. The results trace an evidence ladder: executable artifacts are the only major type where 15 of the 24 datasets are both graded real-world and carry labels that received an independent check, while code-sample datasets-the largest category in both the auto-tagged corpus and the anchor set-are the least realistic: 20 of the 41 draw their vulnerabilities from authentic projects or CVEs, but only 3 keep the sample at the unit the code is deployed in, and only 2 do both-though these are coarse component tests, and just one code-sample dataset meets the codebook's stricter full-context real-world grade. Among these, leakage goes unaddressed by 49 of the 90 datasets where it applies, more than a quarter say nothing about availability, reasoning data has arrived only recently and is mostly model-generated, and primary trace corpora remain limited to three datasets, the total after a corpus-wide screen and a full-text check of every candidate it surfaced, with further datasets releasing traces secondarily behind benchmarks and harnesses.

0.17

CESBench: Benchmarking Large Language Models on Cryptographic Engineering Security for IoT Devices

cs.CR cs.AI ·2026-09-18 ·Wenquan Zhou, An Wang, Jing Liang, Peien Feng, Jingqi Zhang +2 more
benchmark similarity 0.60
Abstract

For Internet of Things (IoT) devices, a secure algorithm alone is not enough: an attacker with physical access can attack the implementation directly, and its flaws are hard to fix once deployed. Large language models (LLMs) are now used to build and analyze such implementations. LLM benchmarks exist for cryptography and general cybersecurity, but none covers cryptographic engineering. In this paper, we present CESBench, 380 expert-written items across six sub-domains of cryptographic engineering security for IoT devices: side-channel, fault injection, implementation, countermeasures, evaluation, and integration. Four task types target different competences: 209 multiple-choice items test recall, 67 judgment items require a security verdict and its justification, 63 scenario items require an engineering diagnosis, and 41 code tasks are graded by 572 test cases. To validate the benchmark, 11 open-weight and proprietary LLMs answer every item. Multiple-choice and code responses are scored automatically, and judgment and scenario responses by an LLM judge, whose scores are checked against a second judge from another model family and human re-scoring. Composite scores range from 54.4% to 83.6%. The top score on each task type is 98.6% for multiple choice, 95.1% for code, and 88.4% for scenario diagnosis, but only 58.8% for judgment. Across models, 88.5% of verdicts are correct, yet their justifications earn only 53.4% of the rubric marks. Multiple choice is near its ceiling for the strongest models and most code tasks are solved, whereas justifying a security verdict remains the weakest competence. The benchmark, prompts, and per-item results are public.

0.20

ReproBench: Benchmarking LLM Agents on Reproducing Vulnerability From Scratch

cs.CR ·2026-09-28 ·Liang He, Sheng Wu, Haomiao Hao, Hongduo Zhao, Jia Yan +1 more
benchmark similarity 0.60
Abstract · 15 pages (9 pages main text + 6 pages appendix), 5 figures, 11 tables. Submitted to AAAI 2027

Large language model (LLM) agents are increasingly evaluated on cybersecurity tasks such as vulnerability reproduction, exploitation, and patching. However, existing cybersecurity benchmarks predominantly operate under a post-environment evaluation paradigm, i.e., handing the agent source code, a container, or an executable binary. This setup bypasses the critical environment reconstruction step, leaving a fundamental question for real-world vulnerability analysis: can an agent autonomously reconstruct the required execution environment and reproduce a vulnerability entirely from scratch? To address this gap, we present ReproBench, an evidence-grounded benchmark designed to evaluate agent capabilities in end-to-end vulnerability reproduction starting from solely a CVE identifier. ReproBench decomposes the full reproduction workflow into six distinct phases, and assesses performance on each phase independently using verifiable experimental artifacts: downloaded firmware images, unpacked binaries, granular analysis logs, and validated crash samples, among others. We instantiate ReproBench with 30 real-world IoT firmware vulnerabilities, which serve as ideal test cases for our from-scratch evaluation setting. Our evaluation demonstrates that 45.3% of test runs resort to vulnerability simulation - a prevalent remediation workaround adopted across all evaluated LLM agents - while only 5.3% of CVE-model pairs achieve successful reproduction of real-world vulnerabilities. Despite the low overall success rate, these non-trivial successful cases confirm that state-of-the-art LLM agents already possess the capacity for fully autonomous end-to-end vulnerability reproduction. Concurrently, our in-depth analysis of failed cases identifies core bottlenecks impeding LLM agents throughout the reproduction pipeline, offering actionable insights for subsequent research.

0.41

VulDec: Vulnerability Detection in Binaries via Neural Decompilation

cs.CR cs.LGcs.SE ·2026-10-24 ·Francesco Crescenzo Grasso, Pietro Liguori, Roberto Natella, Domenico Cotroneo
code/data similarity 0.59
Abstract · Submitted to Automated Software Engineering

Automated vulnerability detection in binary executables is challenging due to the lack of source code and the limited semantic information available at the assembly level. Despite significant progress in machine-learning-based code analysis, the field still lacks methodologies and large, high-quality datasets that provide vulnerable code jointly at assembly- and source-level with precise vulnerability labels. To address this gap, we introduce VulDec, consisting of (i) a new large-scale, carefully curated dataset aligning real-world Assembly-C function pairs with detailed CWE annotations across multiple architectures, and (ii) the first end-to-end framework that leverages neural decompilation for binary vulnerability detection. Tuned LLM-based neural decompilers trained on our dataset achieve strong source-level recovery across architectures (Edit Similarity of approximately 56%, BLEU-4 of approximately 43%), while our tuned LLM-based detectors reach a macro F1-score of 0.61 on an in-distribution realistic test set (3% vulnerability rate). Our findings provide evidence that neural decompilation offers a powerful alternative to conventional low-level binary representations, facilitating robust binary vulnerability detection.

0.35

ICS Cybersecurity Datasets: A Systematic Meta-Review of Coverage, Evaluation Practice, and Structural Gaps

cs.CR ·2026-08-25 ·Konstantinos E. Kampourakis, Vyron Kampourakis, Georgios Kambourakis, Sokratis Katsikas, Stefanos Gritzalis +2 more
similarity 0.59
Abstract

Intrusion detection research in Industrial Control Systems (ICS) heavily depends on public datasets, yet no prior work has systematically assessed whether the collective dataset corpus supports current evaluation claims. This paper addresses this gap through a meta-review of 18 studies between 2019 and 2026, from which 83 ICS, or ICS directly related, cybersecurity datasets are identified, harmonised, and characterised using a unified five-dimensional taxonomy. The taxonomy reveals that the corpus is structurally skewed: 85.5% of datasets concentrate on late-stage OT Disruption tactics, cross-stage IT/OT progression sequences are present in only 8.4% of cases, field-device evidence at Level 0 of the Purdue hierarchy is effectively absent, and operationally sourced data accounts for only 15.7% of the collection. A parallel audit of evaluation practices shows that zero report streaming evaluation, fewer than half apply disciplined train/test partitioning, and only two satisfy reproducibility requirements. Furthermore, a taxonomy-evaluation coupling analysis shows that dataset imbalances constrain the scope and feasibility of several evaluation practices. Based on these findings, we identify three structural imbalances: i) architectural shallowness, ii) progression compression, and iii) cross-domain substitution, and derive a coordinated research agenda which covers cross-stage corpus construction, temporally structured benchmarking, event-level label standards, and governance frameworks for operational data sharing.

0.42