CS Paper Radar

A Set-Theoretic Evaluation Framework for Assessing Asset Administration Shell Instances: Towards Comparability and Suitability

Carsten Ellwein, David Dietrich, Rozana Cvitkovic, Bastian Lang, Jessica Roth, Hansjoerg Tutsch, Andreas Wortmann

cs.SE eess.SY ·submitted 2026-09-15 12:07 UTC · v2

Abstract

Asset Administration Shells (AAS) provide a standardized means of representing assets and their information in manufacturing and increasingly serve as a basis for software services. However, different AAS instances vary in structure, content, and degree of completion, making it difficult to determine whether a given AAS is suitable for a specific application. This paper presents two complementary methods to support the comparison and application-oriented assessment of AAS. First, set-theoretic operations are employed to compare AAS models, enabling the identification of common, missing, and differing submodels and parameters. Second, an AAS suitability model assesses the conformity of an AAS to the requirements of a specific use case. The assessment considers structural conformity, semantic consistency, cardinality, and specification conformity and can be performed either against a reference AAS or a set of required SemanticIDs. A suitability value is derived from the identified deviations and is complemented by a detailed report of missing or non-conforming information. The proposed approach support practitioners and researchers in the comparison of evolving AAS and provide application-specific information on their suitability for manufacturing software services.

Closest papers

Nearest neighbours by meaning across the whole index.

Quality Metrics for LLM-Generated Asset Administration Shells: A Perturbation-Based Evaluation Approach

cs.SE cs.AI ·2026-09-07 ·Janek Groß, Elena Zentgraf, Jens Heidrich
published similarity 0.66
Abstract

The rapid digital transformation of manufacturing, often referred to as Industry 4.0, relies on seamless interoperability between physical and software assets. A central enabler is the Asset Administration Shell (AAS), a standardized digital representation of such assets. Recent advances in large language models (LLMs) enable the generation of AAS submodels from unstructured sources such as product datasheets but raise challenges for quality assurance. In particular, unexpected errors, the lack of ground truth references, and the absence of standardized quality metrics hinder reliable adoption. In this work, we evaluate quality metrics for AI-generated AAS using a perturbation-based evaluation framework. By systematically degrading AAS generation along multiple dimensions, we assess how well different metrics reflect quality changes. Based on a dataset of 200 products from multiple manufacturers, we generate 6,400 AAS instances using GPT-4o-mini, Qwen3, and DeepSeek-R1. Our results show that metrics based on exact matching of property names and similarity-based soft matching of property values, in particular value-based recall and name-based F1 score, provide the most reliable indicators of quality degradation. Furthermore, we quantify the impact of different perturbation types and analyze differences across model families and product segments. These findings support the selection of suitable metrics, the tuning of LLM-based pipelines, and the integration of AI-generated AAS into industrial applications.

0.71

Towards an Asset Administration Shell Maturity Model

cs.SE eess.SY ·2026-09-15 ·Carsten Ellwein, David Dietrich, Rozana Cvitkovic, Andreas Wortmann
top venuecode/data similarity 0.59
Abstract

The Asset Administration Shell (AAS) is increasingly recognized as a fundamental model for the realization of and data exchange between digital twins in manufacturing. An AAS defines a hierarchical data structure to represent any type of asset throughout its entire lifecycle. In the context of AAS-based systems, comparing different AAS instances constitutes a practical challenge, as neither a widely accepted methodological framework nor a maturity model are available to systematically support such analyses. To address this gap, we propose a novel concept of AAS maturity that characterizes the extent to which established digital twin criteria are met and thus enabling comparability of AAS instances. The concepts are derived from the literature and applied through exemplification. These emerging results enable practitioners and researchers to systematically compare AAS instances and support the identification and assessment of further development steps in the digital twin engineering process.

0.81

From Industrial Product Documentation to Automated Digital Product Passport Generation: An Experimental Study of Large Language Models

cs.SE cs.DB ·2026-11-18 ·Marco Buecheler, Grant Richards, Nathan Hartman
similarity 0.57
Abstract · 12 pages, 8 figures. Submitted to NAMRC 55 (2027)

Manufacturing companies that are planning to sell certain products in the European Union (EU) will be required to provide mandatory digital product passports (DPPs). The use of DPPs is to provide standardized product information throughout multiple product life cycles in a circular economy. Required passport information, such as technical specifications, material composition, and environmental product footprint, is available in manufacturers' data sources but is scattered across different heterogeneous systems, documents, and data formats. The given paper investigates using large language models (LLMs) to automate the generation of DPPs based on the asset administration shell (AAS). A retrieval-augmented generation (RAG) approach is used to extract product information from industrial documentation and generate populated AAS instances. Product documentation is processed and divided into text chunks. The text chunks are then stored in a persistent ChromaDB vector database using vector embeddings. The retrieval process combines semantic and lexical search to identify relevant information based on standardized AAS submodel template requirements. The retrieved information is provided to five LLMs to generate structured AAS instances. The experiments cover different industrial products and AAS submodels. Four levels of prompt guidance are investigated to assess their influence on generation quality. The generated passports are evaluated against a ground truth. The experimental evaluation includes 4,560 executions across 76 product-submodel combinations. Of the generated passport outputs, 97.4% are valid JSON, while 44.9% pass the full AAS validation procedure. Under deterministic scoring, mean completeness, conditional accuracy, and overall field correctness are 49.6%, 58.3%, and 30.8%, respectively. An LLM-as-a-judge yields better scores of 66.5%, 88.9%, and 60.4% for the same outputs.

0.20

AAS-RAIL: Improving Information Extraction for Asset Administration Shells through Retrieval-Augmented In-Context Learning

cs.AI ·2026-09-07 ·Janek Groß, Jens Heidrich
similarity 0.54
Abstract

The Asset Administration Shell (AAS) is a cornerstone of Industry 4.0 and the Digital Product Passport, providing standardized digital representations of industrial assets. While manufacturers already maintain extensive technical product documentation, generating AAS instances from existing product datasheets remains a labor-intensive task because technical information is extracted from heterogeneous document structures and often involves company-specific terminology and conventions. In this work, we present AAS-RAIL, a retrieval-augmented information extraction (IE) approach that automatically generates Asset Administration Shells from PDF product datasheets using large language models (LLMs). Instead of relying on a fixed set of few-shot examples, the proposed retrieval-augmented in-context learning (RAIL) approach retrieves LLM-generated extraction helpers from similar Asset Administration Shells to provide instance-specific in-context learning (ICL). This enables the model to adapt its extraction behavior to company-specific naming conventions and formatting styles without fine-tuning. Our core contribution is the dynamic selection of company-specific AAS examples for each datasheet, replacing static prompting with an extraction pipeline that adapts to instances and combines semantic retrieval and structured information extraction. The proposed approach is evaluated on a collection of industrial product datasheets using a selection of open- and closed-weight LLMs. Experimental results show that RAIL consistently improves extraction quality over conventional few-shot prompting, yielding relative improvements of 30.4-52.4%. These results demonstrate that our approach provides an effective improvement for company-specific AAS generation.

0.22

Integrating MBSE Principles with AI to Improve the Quality of Requirements in SMEs Context

cs.SE ·2026-11-06 ·Iryna Volosheniuk, Claude Baron, Jean-Michel Bruel
published similarity 0.53
Abstract

Model-Based Systems Engineering (MBSE) is a promising approach to improve systems engineering practices, particularly by enhancing requirement management, traceability, and early validation and verification (V\&V). However, its adoption remains challenging, especially for small and medium-sized enterprises (SMEs), which often lack structured processes and solutions adapted to their scale and constraints. Our goal is to propose a lightweight MBSE methodology tailored to the needs and constraints of SMEs. This paper focuses specifically on requirements refinement, traceability, and early V\&V. The approach integrates MBSE principles with artificial intelligence-assisted analysis to improve requirements quality while remaining compatible with existing tools and practices. A proof of concept is developed and applied in an industrial case study to evaluate the feasibility of the approach. The results demonstrate how early V\&V activities can be supported within an SME context, while improving traceability across engineering artifacts. These findings highlight the potential of a lightweight, integrated MBSE approach to support the progressive adoption of MBSE in SMEs, thereby improving overall engineering efficiency.

0.39

CRAFT-AI: Capability and Requirement Alignment for Flexible Manufacturing through Agentic AI

cs.AI ·2026-11-09 ·Kiavash Fathi, Hans Wernher van de Venn
similarity 0.51
Abstract · Preprint intended for submission to the 37th European Safety and Reliability Conference (ESREL 2027)

One of the fundamental needs for reaching resilient production infrastructure is flexible manufacturing, which requires translating changing production requirements into implementable system configurations. This paper introduces \textbf{CRAFT-AI}, an agentic AI approach combining natural language (NL) requirement interpretation, Asset Administration Shell (AAS)-based capability descriptions, and reusable engineering knowledge from existing manufacturing configurations. A tool-using AI agent matches requirements with available capabilities and reference implementations to generate manufacturing scenes and corresponding AAS instances. The approach is demonstrated using a Factory I/O prototype with twelve reference scenes and five illustrative manufacturing tasks of varying complexity, providing an initial feasibility assessment of the proposed AAS-enabled, system-level generative engineering approach. For the four valid user task descriptions, proper requirements, correct references to the relevant engineering knowledge, correct components and lastly a valid Factory I/O scene are generated. For the task with missing engineering knowledge and a wrong sensor selection, the user is informed about which requirements of the scene generation lack domain knowledge and also which sensor is not compatible with the selected raw material by the user with link to the related documentation. After fixing the task description, the correct scene is seamlessly generated. \textbf{CRAFT-AI} highlights the importance of safeguards for risk perception and communication in foundation model-driven solutions.

0.16

Developing a Unified Verification and Validation Activity Standard at JPL

cs.SE eess.SY ·2026-09-23 ·Maximilian Vierlboeck, Ellen Van Wyk, Natalia Sanchez, Bogdan Oaida, Torrance James Eberhart +4 more
similarity 0.51
Abstract · 21 pages, 10 figures

Verification and validation practices (V&V) at NASA's Jet Propulsion Laboratory (JPL) have diverged over the past decade, creating fragmentation that increases overhead, reduces cross-project efficiencies, and inhibits institutional knowledge transfer. We present a unified V&V activity schema developed through human-centered design workshops involving 29 practitioners across multiple mission types and disciplines. The schema builds on a relationship-based architecture that allows for separating methods (Test, Analysis, Inspection, Demonstration, and Review of Design) while maintaining a common attribute set. Formalized as a platform-agnostic SysML model, the schema defines bidirectional relationships between requirements, V&V activities, venues, and evidence. Implementation in JPL's Jama platform demonstrates controlled customization through templates and modular item types, balancing rigor with agility while enabling automations, pattern reuse, and digital thread integration.

0.30

Comparing Domain-Model Similarity Metrics Against Human Expert Ratings

cs.SE ·2026-08-15 ·Vasiliy Seibert
similarity 0.51
Abstract

Domain models are a primary artefact in model-driven software engineering, where they capture the shared understanding between stakeholders and serve as the contractual basis for downstream software development. Automatic comparison of these semantic models has diverse application areas such as requirements engineering, education, automatic generation of domain models and model reuse and repository mining. The literature offers a variety of presented metrics, but for practitioners there is no defensible way to choose between them. The contribution of this paper is the implementation of five such metrics, their execution on a fixed set of 39 domain-model comparisons and the comparison of each metric's output against the human expert ratings produced for the same comparisons. Two research questions are addressed. RQ1 asks how close, on average, each metric is to the human expert rating across the 39 comparisons. RQ2 asks how consistent each metric's per-comparison distance from the human expert rating is. The findings reveal that no single metric achieves dominance across all criteria; rather, different metrics each yield competitive results on individual criteria - some closest on average, others best preserving the per-pair ordering - which suggests that an ensemble approach combining multiple metrics may serve as a viable substitute for human expert grading. The metric implementations are an artefact of this work and are published in accordance with the FAIR4RS recommendations (DOI: 10.5281/zenodo.20942596).

0.04