CS Paper Radar

SoK: Failure Modes in Common Criteria Product Evaluation - A Taxonomy and Design-for-Evaluability Guidance

Punit Suketu Patel

cs.CR cs.SE ·submitted 2026-10-07 15:09 UTC · v1
publishedsurvey

Abstract

The Common Criteria (CC; ISO/IEC 15408) is the principal international framework for evaluating the security of IT products, and its certificates gate procurement across government, defense, and regulated industry. Yet evaluations routinely stall, fail, or yield certificates whose assurances do not survive real-world deployment. Prior work has approached this from two directions: economic critiques of misaligned incentives and "security theatre"; and recent data-driven studies that quantify vulnerabilities in already-certified products. Comparatively little systematizes the problem from the evaluator's operational vantage - recurring, cross-vendor failures against the Common Criteria work units themselves. This paper presents a systematization of knowledge (SoK) of failure modes in Common Criteria product evaluation. Drawing on the standard, the Common Evaluation Methodology (ISO/IEC 18045), public NIAP Protection Profiles, and published Security Targets and Certification Reports, we organize recurring failures into a lifecycle taxonomy spanning Security Target scoping, Protection Profile conformance, assurance evidence, cryptographic requirements, functional testing, vulnerability analysis, operational guidance, and post-certification configuration drift. We analyze their root causes, derive a design-for-evaluability framework that product teams can apply before evaluation begins, and identify open problems for cloud, continuous-delivery, and AI-enabled systems.

Comment
25 pages, 1 figure, 1 table. Accepted at the Security Standardisation Research (SSR) Conference 2026; to appear in Springer LNCS. Author's submitted version, prior to peer-review revisions

Closest papers

Nearest neighbours by meaning across the whole index.

A Cross-Domain TARA-to-NAC Rule Mapping Methodology: From Automotive OEM Networks to Energy Storage Systems

cs.CR cs.NIeess.SY ·2026-11-06 ·Krishna Teja Medam, Thiago Quinelato
similarity 0.61
Abstract · Preprint, 23 pages, SAE format

Threat analysis and risk assessment under ISO/SAE 21434 ends with cybersecurity goals, but existing methods give little structured guidance for turning those goals into enforceable network access control rules. Compliance standards such as IEC 62443, ISO/SAE 21434, and ISO/IEC 27001 are usually applied as parallel efforts rather than as one enforcement architecture. This paper presents a six-step methodology that maps cybersecurity goals from a threat analysis and risk assessment to network access control rules, each placed at a named enforcement point and traceable to the threat records behind it. It also presents a unified compliance framework that treats IEC 62443 and ISO/IEC 27001 as a fixed substrate and domain-specific regulation as an interchangeable overlay. The methodology is applied to an automotive original equipment manufacturer plant network, addressing multi-tier supplier authorization and legacy operational technology devices, and then to utility-scale battery energy storage, where North American Electric Reliability Corporation Critical Infrastructure Protection standards and ISO/IEC 27019 replace the automotive-specific standard while the substrate stays constant. In the automotive example, 37 threat entries and 24 cybersecurity goals yield 23 rules covering 11 of the 13 in-scope goals; the energy example yields six rules. Three derived rules have no equivalent in the cited standards. Both examples are desk-based: the methodology is applied to them, not tested on them. The result is a reusable pattern for translating risk assessment into enforcement across critical infrastructure sectors, with empirical validation left as future work.

0.29

Preparing an AI-Augmented SIEM for the EU Cyber Resilience Act: A Practitioner Case Study

cs.CR cs.CY ·2026-10-06 ·Georgios Koutidis, Nikolaos Kekatos, Marina Korgiala-Karyda, Alexios Lekidis, Tom Nianios
publishedcode/data similarity 0.61
Abstract · 7 pages, 2 figures, 2 tables. Accepted at the 2026 IEEE International Conference on Cyber Security and Resilience (IEEE CSR 2026)

The EU Cyber Resilience Act (CRA), Regulation (EU) 2024/2847, makes product cybersecurity a lifecycle obligation for products with digital elements on the EU market: risk assessment, vulnerability handling, conformity documentation, and Article 14 incident- and vulnerability-reporting readiness must be operational before market placement. Small and medium-sized enterprises that build security products are doubly exposed, since their products are in scope while their customers expect them to be exemplary. This case study documents a CRA preparedness pilot for one such product, SEUXDR, an AI-augmented security monitoring product with a large-language-model active-response component, on the open-source CYBERFORT platform. We contribute a reproducible six-step recipe (Scope and Classify, Asset Registration, Produce Evidence, Map to CRA, Gap and Actions, Audit Pack), two end-to-end traceability threads, and a pilot snapshot tracing product risks through baseline and AI-specific controls and policies to CRA objectives. It offers practitioners a replicable starting point for translating CRA legal text into operational preparedness for incident response, vulnerability reporting, and conformity assessment.

0.40

CYBERFORT: A Compliance-Chain Platform Operationalising the Cyber Resilience Act for SMEs

cs.CR cs.CYcs.SE ·2026-10-07 ·Nikolaos Kekatos, Georgios Koutidis, Mihaela Curcă, Angelos Fountoulakis, Marios Ioannou +6 more
publishedcode/data similarity 0.61
Abstract · 7 pages, 3 figures, 2 tables. Accepted at the 2026 IEEE International Conference on Cyber Security and Resilience (IEEE CSR 2026)

The EU Cyber Resilience Act (CRA) turns product cybersecurity into a lifecycle compliance obligation for manufacturers, importers, distributors, and integrators of products with digital elements on the EU market, a load that falls largely on small and medium-sized enterprises (SMEs) that rarely have dedicated governance, risk, and compliance (GRC) capacity. We present CYBERFORT, an open-source CRA-first compliance platform developed under the EU Digital Europe Programme and one of twelve projects in the EU CRA cluster. CYBERFORT operationalises the CRA through a guided scope self-assessment, a question bank tied to Annex I and the vulnerability-handling obligations, and a compliance-checking engine that links every answer to controls, policies, and machine-attested evidence, reusing ISO/IEC 27001, NIS2, and GDPR controls only where they coincide with CRA obligations. Its central contribution is the compliance chain, a traceable structure linking each product risk through its controls and policies to the CRA obligations it satisfies, and onward through evidence to the technical-documentation file and EU declaration of conformity, so that every operational gap is traceable and can be closed before market placement. Deployed at https://access.cyber-fort.eu/ for a first cohort of 43 organisations, the platform is presented with the engineering behind the chain, measured results from a completed end-to-end case study on a SIEM/XDR product with AI-driven remediation spanning the CRA obligation chapters, and the controlled effort study that remains in progress.

0.64

Non-Degenerate Risk Certification for Automated Security Decisions: A Decision-Contract Theory with ATT\&CK-Aligned Triage as a Worked Instance

cs.CR cs.LG ·2026-08-12 ·Zhenpeng Li
similarity 0.61
Abstract · 17 pages, 2 figures, 10 tables

An unconditional risk bound on automated decisions can be satisfied without automating anything, since a selector that never acts drives the bound to zero. We show this is structural: any risk certificate is defined over a decision contract, the inputs a system acts on plus the semantic relation under which an output counts correct, and weakening either hides base-classifier error. We develop a decision-contract theory: an error-conservation law showing error is only reassigned among harmful automation, human deferral, and semantic masking; a label-free singleton capacity certifying structural incapacity, with a risk-feasible refinement separating recoverable threshold misalignment from risk-constrained incapacity; and a non-degenerate actionability certificate excluding all-abstain solutions by construction. We instantiate this on ATT\&CK-aligned alert triage for LLM-based intrusion detection, the setting that exposed the vacuity failure. Across 3 IDS datasets, 6 LLMs, and 4 error-rate thresholds, empirical false-attribution risk stays at or below target in 90.3% of configurations, with 83.4% mean correct automation. The capacity diagnostic explains every low-utility configuration; its refinement separates genuine misalignment from risk-constrained incapacity, confirmed by an exhibited alternative threshold; a training-stability re-run finds no confirmed structural-incapacity instance; and real fine-grained attack-subtype labels confirm the coarsening-transfer identity under a genuine many-to-one map, with small but non-zero masking mass.

0.10

A Failure Mode Taxonomy and Severity Model for Ethereum Smart Contracts

cs.SE ·2026-10-30 ·Fernando Richter Vidal, Naghmeh Ivaki, Nuno Laranjeiro
survey similarity 0.61
Abstract

Smart contracts are a core component of blockchain systems, but their immutability makes failures in deployed code catastrophic and irreversible. Existing work classifies vulnerabilities, defects, and attacks, yet the observable failures of smart contracts, the deviations from intended behavior or the expected ledger state seen once a contract is deployed, have been largely disregarded. In this paper, we design and validate a failure mode taxonomy for Ethereum smart contracts, together with a severity model that ranks failures by their observable impact. Using a literature review as input, we follow the taxonomy development method of Nickerson et al. to derive the failure modes and their dimensions, and we subject the resulting taxonomy to a saturation exercise over a systematic literature review, the OpenSCV vulnerability catalog, and smart contract incidents reported after the search cutoff. Each failure mode is characterized along five dimensions, namely root cause, violated security property, impact (recorded as primary, secondary, and other), affected component, and nature, and is given a standardized name targeting the observable malfunction rather than the underlying vulnerability. From the taxonomy, we derive CIPER, a five-level severity model that groups failure modes by their observable impact. The saturation exercise produced no new failure modes or dimensions. The taxonomy offers a shared vocabulary for describing what smart contracts do wrong when they fail, and CIPER links that vocabulary to the severity of each observable outcome.

0.25

Who Assures the Verifier? An Executable Assurance-Locus Audit of the European Digital Identity Wallet

cs.CR cs.CY ·2026-08-11 ·Anton Sokolov
code/data similarity 0.60
Abstract · 11 pages, 1 figure, 2 tables. Technical pre-results version. Includes an ancillary reproducibility bundle with a 17-rule profile, 36 frozen …

The European Digital Identity Wallet (EUDI Wallet) architecture places material duties on relying parties: they register services and intended uses, authenticate to Wallet Units, validate presentations and trust anchors, and make risk-based status decisions. Wallet certification and the emerging Functional Conformance Assessment Framework provide increasingly structured wallet-side evidence. A different question remains: what independently rerunnable evidence shows that the concrete relying-party verifier version used in a transaction enforced the applicable request, presentation and reliance-decision controls? We conduct an assurance-locus audit of current law, Architecture and Reference Framework (ARF) 3.0.0, ETSI metadata, FCAF scope and three pinned open-source verifier codebases. We then design a 17-rule research profile, a machine-readable evidence receipt and 36 frozen synthetic transactions. Three heterogeneous study-authored implementations (Python, JavaScript and jq) execute 108 cases with zero oracle mismatches and zero cross-path disagreements. The experiment establishes determinism and implementability of the proposed decision model, not product conformance or certification. Source inspection finds substantial protocol-verification mechanisms in all three public codebases but no single audited evidence object joining RP registration and purpose, exact verifier/policy version, transaction verdict and downstream attribute use. We therefore propose an RP-as-system-under-test evidence unit that complements, rather than displaces, wallet certification, registration and supervision. A preregistered census of 26 appointed experts is prepared to test content validity and governance feasibility; recruitment awaits the applicable ethics/data-protection determination.

0.26

ARC-2: Continuous Compliance Through Delta Management

cs.RO cs.AIcs.CY ·2026-11-16 ·Tord Johan Eide, Einar Holt
similarity 0.59
Abstract · 22 pages. Conceptual framework for continuous compliance monitoring, behavioral drift detection, and operational authorization of autonomous…

ARC 1 introduced a three-layer governance architecture for autonomous systems comprising model safety validation, cognitive certification (CRAB), and operational authorization (LEA). However, it did not address how compliance should be maintained after deployment. This paper introduces Delta Management, a continuous governance framework that measures deviations between a system's current operational state and its certified baseline, defined as a region rather than a fixed point. Compliance is assessed through continuous monitoring instead of static certification. Four extensions are proposed. First, CRAB evaluation incorporates the worst verified behavioral deviations, with isolated outliers quarantined pending confirmation to prevent unjustified restrictions. Second, a layered trigger taxonomy distinguishes hard safety violations, critical single-dimension failures, correlated multi-system degradation, and risk-weighted composite indicators, each triggering proportionate responses. Third, a graduated safe transition protocol separates withdrawal of operational authorization from physical shutdown, consistent with established safety standards and EU AI Act requirements. Fourth, environmental delta is introduced as a distinct baseline validity condition: environmental drift beyond the certified Operational Design Domain (ODD) triggers recertification or ODD revision rather than automatic authority contraction. Together, these extensions transform ARC from a deployment-time certification architecture into a continuous framework for monitoring, evaluating, and enforcing compliance throughout the operational lifecycle of autonomous systems. The framework addresses the continuous compliance obligations established by EU AI Act Article 72(2).

0.20

AspisAI: A Canonical, Machine-Interpretable Governance Framework for Automated Multi-Standard Compliance Monitoring

cs.CR cs.CYcs.SE ·2026-09-09 ·Tsafac Nkombong Regine Cyrille, Hasan Dag, Reiner Creutzburg, Knut Haufe
publishedcode/data similarity 0.58
Abstract · 6 pages, 2 figures. Accepted at IACyC 2026, the 2nd CyberMACS International Applied Cybersecurity Conference

Organisations operating in regulated and critical-infrastructure sectors must satisfy multiple, heterogeneous cybersecurity and privacy instruments simultaneously, including but not limited to ISO/IEC~27001, the NIST Cybersecurity Framework~2.0, Cyber Essentials, and the GDPR. In practice, these obligations are managed through manual mappings, spreadsheet-based tracking, and periodic audits that are costly to maintain, inconsistent across standards, and weak in traceability. This paper presents \emph{AspisAI}, a bounded, standard-agnostic governance framework that translates selected requirements from several frameworks into a canonical, machine-interpretable control model, and evaluates submitted evidence against condition-based decision rules to produce explainable, traceable compliance determinations. Within a bounded scope of 26 representative requirements, the framework is evaluated in a controlled simulation against five governance-oriented criteria and, critically, against two external reference points that mitigate the circularity of single-author evaluation: its cross-standard mappings are validated against NIST's own published informative references, with 57\,\% exact agreement and divergences confined to same-family controls, and the framework is applied to real third-party evidence from the OpenSSF Scorecard, surfacing genuine governance gaps in a live open-source project. The controlled results, comprising full requirement encoding, 88.5\,\% mapping coverage, complete traceability, and correct detection of all introduced gaps, establish functional correctness, while the external validation provides evidence of applicability beyond the simulation. The contribution is therefore a demonstration that a canonical, provenance-preserving governance model can render multi-standard compliance both automatable and auditable.

0.49