CS Paper Radar

ICS Cybersecurity Datasets: A Systematic Meta-Review of Coverage, Evaluation Practice, and Structural Gaps

Konstantinos E. Kampourakis, Vyron Kampourakis, Georgios Kambourakis, Sokratis Katsikas, Stefanos Gritzalis, Mario Rodríguez-Béjar, José Luis Hernández-Ramos

cs.CR ·submitted 2026-08-25 15:56 UTC · v1

Abstract

Intrusion detection research in Industrial Control Systems (ICS) heavily depends on public datasets, yet no prior work has systematically assessed whether the collective dataset corpus supports current evaluation claims. This paper addresses this gap through a meta-review of 18 studies between 2019 and 2026, from which 83 ICS, or ICS directly related, cybersecurity datasets are identified, harmonised, and characterised using a unified five-dimensional taxonomy. The taxonomy reveals that the corpus is structurally skewed: 85.5% of datasets concentrate on late-stage OT Disruption tactics, cross-stage IT/OT progression sequences are present in only 8.4% of cases, field-device evidence at Level 0 of the Purdue hierarchy is effectively absent, and operationally sourced data accounts for only 15.7% of the collection. A parallel audit of evaluation practices shows that zero report streaming evaluation, fewer than half apply disciplined train/test partitioning, and only two satisfy reproducibility requirements. Furthermore, a taxonomy-evaluation coupling analysis shows that dataset imbalances constrain the scope and feasibility of several evaluation practices. Based on these findings, we identify three structural imbalances: i) architectural shallowness, ii) progression compression, and iii) cross-domain substitution, and derive a coordinated research agenda which covers cross-stage corpus construction, temporally structured benchmarking, event-level label standards, and governance frameworks for operational data sharing.

Closest papers

Nearest neighbours by meaning across the whole index.

The Cybersecurity Dataset Observatory: Which Data for AI in Cybersecurity?

cs.CR cs.AI ·2026-11-16 ·Francesco Cerasuolo, Gabriele Mangiacapre, Antonio Pescapè
benchmarkdatasetcode/data similarity 0.72
Abstract

AI is increasingly used for cybersecurity tasks such as intrusion and malware detection, in enterprise, industrial, vehicular and mobile networks. Its effectiveness depends on data: models need representative network traffic, both to be trained and to be evaluated realistically. Public datasets of cybersecurity network traffic, however, are scattered across laboratory pages, repositories and aggregators, and are continuously released, corrected and withdrawn: 92 of the 234 datasets we catalog were released in 2023 or later. At this pace, surveys that list them are outdated soon after publication, since they describe the field only as it was when they were written. We present the Cybersecurity dataset observatory, a public and continuously updated catalog of cybersecurity datasets based on network traffic. New datasets are added as they are found or submitted, every record is periodically re-checked against its original source, to which it links rather than hosting the data, and every change is recorded in a public change log. This paper documents the first release: we define a traffic-centered scope and a seven-axis taxonomy, and review 234 datasets (204 original captures and 30 derivatives) across 14 domains. The review shows a field that grows quickly but unevenly, where 91 datasets distribute only processed records, a few laboratories produce a large share of the data, and UAV, wireless and V2X networks have little public evidence, while 5G has few real captures. We then relate the catalog to common uses in AI research, outline a procedure for selecting datasets, and discuss benchmark pitfalls and open gaps.

0.35

Recording-Aware Evaluation of Multimodal OT Intrusion Detection

cs.CR cs.LG ·2026-10-16 ·Mustafa Umut Ozbek, Khalil El-Khatib, Li Yang
publishedbenchmarkcode/data similarity 0.69
Abstract · 12 pages, 3 figures, 5 tables. Accepted for presentation at the Industrial Control System Security Workshop (ICSS 2026), held in conjunction…

Machine-learning-based intrusion detection systems (IDSs) for operational technology (OT) are commonly evaluated on benchmark datasets in which normal and attack data are collected in separate recordings. In such settings, high performance may reflect recording-associated differences as well as attack behavior. We introduce a recording-aware evaluation procedure that compares attack windows with background from the same recording, uses prior data for model fitting and calibration, tests temporal alignment against randomized event schedules, and reports event coverage together with alarm burden. On the ProvICS multimodal dataset, cyber modalities strongly distinguish the benign and attack recordings before the first labeled attack. Same-recording evaluation reverses the apparent modality ranking across all studied window sizes: the physical view becomes the strongest individual modality, while cyber-only fusion shows the largest difference between cross-recording and same-recording AUC, reaching 0.54. Operational analysis further shows that permissive event-overlap metrics can overstate detection when a small number of long alarm episodes cover multiple actions. An external analysis on HAI shows smaller and reference-dependent recording sensitivity. These findings show that cross-recording benchmark performance should be interpreted alongside same-recording, forward, temporal, and operational evaluations.

0.44

A Case-Control Measurement Study of OSINT Source Effectiveness for Critical Infrastructure Defense

cs.CR ·2026-08-21 ·Ekrem E. Emeksiz, Jeel Piyushkumar Khatiwala, Divyangkumar Patel, Weifeng Xu
publishedcode/data similarity 0.68
Abstract · 10 pages, 3 figures, 5 tables. Accepted at IEEE Conference on Communications and Network Security (CNS) 2026. Authors' full-length version; …

Defenders of critical infrastructure (CI) subscribe to many public open-source intelligence (OSINT) feeds without an empirical basis for which feeds actually precede attacks. We provide one. Across 54 confirmed CI cyberattacks from 2010 through 2024 spanning twelve named CI sectors plus a cross-sector category (consolidation rules in Section IV), paired with 12 null-control vulnerability cases drawn from the same source space, we audit per-source attack coverage, null-case contamination, and signal lead time for ten public OSINT source classes that meet a minimum-volume threshold. Sources separate cleanly into three operationally distinct mission profiles (pooled Fisher exact p = 3.4x10^-8): precursor (six classes with zero observed null firings at coverage at or above 5%), disclosure-exposure (three classes whose null contamination meets or exceeds attack coverage), and one large broad-coverage class that mixes the two profiles but retains 91.3% within-corpus precision. The precision-side classification is stable across a 2019 temporal partition and across a US-versus-non-US geographic partition. Two sources, one broad-coverage and one precursor, cover 92.6% of corpus attacks; three cover 96.3%. The greedy portfolio at k = 3 outperforms the mean random three-source subset by 39.8 percentage points. Several source classes widely treated as canonical for industrial control system defense fall into the disclosure-exposure profile by operational mission, not by quality. Per-sector, per-actor, and per-jurisdiction portfolios diverge in rank order despite a shared rank-one source. The corpus, linkage protocol, and classification rules are released.

0.66

A Roadmap to Available ICS Datasets and Testbeds for Cybersecurity Research

cs.CR eess.SY ·2026-08-31 ·Ebtesam J. Alqahtani, Mohammad Hammoudeh
similarity 0.67
Abstract

Industrial Control Systems (ICS) are the backbone of many critical infrastructure sectors; however, their growing level of connectivity, long lifespan and integration with the Information Technology (IT) environment introduces numerous cybersecurity challenges. The merging of Operational Technology (OT) and IT along with the deployment of Industry 4.0 technologies increases the attack surface of ICS environments, which in turn makes them more vulnerable to advanced cyber threats. Therefore, many researchers have shown interest in the field of cybersecurity of ICS. The topics of intrusion detection, anomaly detection, threat intelligence, attack simulation and resilience assessment of ICS have received much attention. Nevertheless, the development and testing of cybersecurity solutions for ICS remains to be challenging due to the lack of appropriate datasets and experimental environment. The main objective of this paper is to provide the roadmap of existing ICS cybersecurity datasets, testbeds and digital twins. This paper presents various taxonomies along with systematic analysis of architecture, characteristics, capabilities, pros and cons of these tools. The results of the analysis demonstrate the presence of persistent problems such as lack of standardized benchmarking datasets, lack of modern attack scenarios, insufficient number of datasets based on real operational traffic and difficulty in validating artificial intelligence-driven cybersecurity solutions. In addition to summarizing current research on ICS cybersecurity datasets and testbeds, this roadmap provides the identification of research gaps and recommendations on creation of new tools.

0.20

Polar: LLM-Powered Synthesis of Real-World Cyber Evidence for Prioritization and Mitigation

cs.CR cs.AI ·2026-10-05 ·Luoxi Tang, Yuqiao Meng, Ankita Patra, Weicheng Ma, Muchao Ye +1 more
similarity 0.66
Abstract

Cyber threat analysis increasingly depends on evidence distributed across vendor advisories, vulnerability databases, and threat intelligence sources. Turning these fragmented observations into timely decisions requires models to connect technical severity with evolving exploitation evidence and available defensive actions. We present POLAR, an LLM-powered framework for synthesizing real-world cyber evidence into threat-centric assessments for prioritization and mitigation. POLAR first disentangles overlapping incidents and grounds each threat in source-linked evidence. For prioritization, it infers severity metrics from cyber evidence and combines the resulting assessment with temporally ordered exploitation signals to estimate near-term exploitation likelihood. For mitigation, it links the synthesized threat data to authoritative remediation knowledge and organizes applicable actions according to threat urgency and operational constraints. We evaluate POLAR on real-world vulnerability evidence collected from public resources and compare it with multiple baselines. Across heterogeneous incidents and zero-day settings, POLAR improves threat ranking and mitigation retrieval while producing evidence-linked intermediate assessments that support analyst inspection. The results establish evidence synthesis as a practical foundation for LLM-based cyber decision support across related security tasks.

0.12

Research Methodologies for Cybersecurity in Enterprise Environments: A Narrative Review, Synthesis and Executable Guide

cs.CR ·2026-08-25 ·Tran Duc Le
surveyposition similarity 0.65
Abstract · 31 pages, 16 figures, 4 tables

Enterprise cybersecurity research draws on a wider range of methods than any single community routinely teaches. Researchers face a selection problem before they face a technical one: a study may simultaneously need a systematic review, a design-science artifact, a controlled detection experiment, an interview study, or an attack-graph model. This paper addresses that problem in two ways. First, it provides a narrative review and synthesis of methodological practices across a verified corpus of 151 works. We organise these practices into eleven methodology families, detailing for each what questions it answers, the strength of its supporting evidence, and its common failure modes. Second, we convert each family into an executable protocol comprising ordered steps, required instruments, evaluation criteria, common validity threats, and a reporting checklist. Every protocol is also visually mapped to make the sequence, decisions, and threats legible at a glance. We also treat contradictions in the literature as evidence. For example, reported rankings of intrusion-detection algorithms are wildly inconsistent across individually careful studies. We argue this pattern is most parsimoniously explained by variations in evaluation design rather than the algorithms themselves, as these studies differ in design dimensions known to shift results by more than the margins separating the algorithms. Ultimately, the evidence supports methodological pluralism disciplined by explicit validity reasoning. We conclude that researchers must match their evaluation design to the decision under study, triangulate technical against organisational evidence, explicitly state the population a result generalises to, and report the conditions under which the result would not hold.

0.36

Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks

cs.CR cs.AIcs.CL ·2026-09-08 ·Aymene Berriche, Cathrine Shalby, Mohannad Alhanahnah, Yazan Boshmaf
benchmark similarity 0.64
Abstract

Large language model (LLM) benchmarks are often treated as fixed datasets with stable scores, yet their outcomes depend on configurable evaluation pipelines. We audit eight cybersecurity benchmarks across 10 proprietary, open-weight, and cybersecurity-specialized LLMs. By modeling benchmarks as measurement pipelines, we identify 15 systematic failure modes and show that a single pipeline choice can change a model's score by more than 80 percentage points and substantially alter model rankings. At the cross-benchmark level, two semantically similar task pairs rank the same models differently because of incompatible evaluation conventions. Under an evaluation harness that standardizes pipeline choices while preserving task semantics, nine of 10 models shift by at least three ranks on at least one benchmark. These results show that cybersecurity LLM benchmark scores are pipeline-dependent and motivate pipeline-aware auditing as a core requirement for reliable model evaluation.

0.17

SecLog-Bench: A Scalable Benchmark for Cybersecurity Log Reasoning with Scaffolded Evidence

cs.CR cs.AI ·2026-10-22 ·Yifan Wang, Dongliang Xu, Xiaoyong Li, Jing Li, Peter S. Pan +1 more
benchmarkcode/data similarity 0.64
Abstract · 17 pages, 9 figures. Submitted to IEEE Transactions on Dependable and Secure Computing. Code and data: https://github.com/SecLog-Open/SecLog

Large language models (LLMs) have shown strong potential for cybersecurity, leading to growing interest in LLM-based automated cybersecurity log analysis. However, current datasets (1) are limited in scale; (2) rely mainly on single-source evidence; and (3) have only binary or multi-class attack labels, without dense annotations of the reasoning process. To address these limitations, SecLog-Bench has three key features. First, we build an extensible simulation platform that covers 29 attack scenarios, supports repeated execution and the integration of new scenarios, and enables scalable generation of cybersecurity logs with attack-stage labels. Second, we use multi-source logs, including audit logs, application logs, and network traffic, to jointly analyze cyber attacks. Third, we annotate each sample with the essential context for cybersecurity log reasoning and a structured step-by-step evidence chain. We evaluate eight mainstream LLMs on this benchmark for post-detection attack-progress reasoning. Results show that, when provided with scaffolded evidence, current LLMs exhibit modest capability in attack-progress reasoning, with the best model achieving only 44.44% accuracy and 45.07% Macro-F1. Their reasoning remains unreliable; i.e., the highest Stage Reasoning F1 (SRF1) score is only 54.87%. Moreover, even correct predictions may be accompanied by flawed reasoning, and reasoning quality is often misaligned with prediction accuracy.

0.35