CS Paper Radar

UnespDataLens-RM: A Reference Model for Analytical Data Engineering with Governance, Quality, Provenance, and Reproducibility

Ronaldo Celso Messias Correia, Douglas Francisquini Toledo, Camila Tolin Santos da Silva

cs.DB ·submitted 2026-09-08 03:15 UTC · v1

Abstract

The growing reliance on data in analytical processes and evidence-based decision-making has reinforced the importance of Data Engineering in building pipelines capable of integrating, transforming, validating, and delivering data from heterogeneous sources. However, the reliability of analytical assets depends not only on data processing capabilities but also on mechanisms for governance, quality assurance, provenance, traceability, versioning, and reproducibility throughout their lifecycle. These responsibilities are commonly addressed by different models, frameworks, and operational practices, resulting in methodological fragmentation across the analytical data lifecycle. To address this gap, this article proposes UnespDataLens-RM, a technology-independent reference model that integrates technical-operational processes and cross-cutting capabilities within a unified structure for Analytical Data Engineering. The model aims to support the specification, organization, and evolution of analytical pipelines by incorporating governance, quality, provenance, traceability, and reproducibility from the design stage. Developed following the Design Science Research approach, UnespDataLens-RM comprises eight technical-operational modules, eight cross-cutting modules, complementary dimensions, and a formalized set of artifacts, metrics, and validation criteria. The resulting specification offers a conceptual and methodological framework for future instantiations and empirical evaluations of analytical pipelines designed to be more governable, documented, traceable, auditable, and reproducible.

Closest papers

Nearest neighbours by meaning across the whole index.

Specification-Driven Data Architecture Reconstruction: From Physical Code to Logical and Conceptual Specifications

cs.LO cs.DBcs.SE ·2026-09-12 ·Oleg Grynets, Olena Pochernina, Vasyl Lyashkevych
similarity 0.66
Abstract · 15 pages, 3 figures, 7 tables, 35 references

Legacy database migrations often begin with incomplete or outdated documentation, leaving physical data definition language (DDL) as the principal evidence of data architecture. However, DDL does not fully encode conceptual intent, and model-generated completions can be plausible without being correct. This study proposes and evaluates a provenance-aware, deterministic-first pipeline for reconstructing logical and conceptual data specifications from Oracle-oriented DDL while explicitly separating observed facts, deterministic derivations, and large language model (LLM) suggestions. The pipeline performs DDL investigation, parsing, consolidation, primary-key backfilling, type normalization, and declared relationship-graph construction before optional LLM-assisted enrichment. It preserves source provenance in the deterministic catalog and declared relationship graph, and records inferred primary-key and foreign-key candidates in a separate reviewable overlay. We evaluated the implementation on 249 artifactized schema samples comprising 1,225 SQL files. The pipeline completed 244 samples (97.99%); 52 completed samples contained no extractable DDL. Across completed samples, the deterministic path reconstructed 208 tables and recovered 278 declared foreign-key records; 168 of the reconstructed tables lacked an explicitly parsed primary key before backfilling. LLM enrichment generated 100 foreign-key candidates in 36 samples, but the parent-table admissibility rate was only 17.9% for logical-specification candidates and 17.5% for conceptual-specification candidates. These findings show that the proposed deterministic-first architecture can preserve an auditable structural baseline, quantify observed primary-key and relationship gaps, and prevent model-generated hypotheses from being silently promoted to source-grounded architectural facts.

0.30

Specification-delta-driven data governance: an empirical study of the «spec-delta» as the unit of change in lakehouse data platforms

cs.AI cs.PL ·2026-08-20 ·Pablo Ramirez Amador
similarity 0.62
Abstract

Spec Driven Development SDD has consolidated the idea that the specification rather than the code should be the primary artefact governing AI assisted work. Tools such as GitHub Spec Kit, and proposals such as Constitutional SDD, have formalised this principle in the software domain, while the executable data-contracts literature has extended it to schema and quality enforcement at run time. Nevertheless, the treatment of the specification delta OpenSpec's core idea that every change should produce a reviewable increment of requirements as the unit of change in data platforms remains empirically unexplored, even though many data-platform changes are contractual (new datasets, service-level agreements, metric semantics, access policies) rather than purely code changes. This work formalises the spec-delta concept, proposes a taxonomy of data platform changes according to their suitability for incremental specification, and defines a controlled experiment comparing a spec-delta-driven workflow against a conventional code pull-request workflow without a delta. The response variables are discovery to deployment time, the density of defects reaching the Silver and Gold lakehouse layers, cross-tool metric divergence, and reviewer cognitive load measured with NASA TLX. The paper explicitly reserves a demonstration-and-laboratory section for instantiation on a real lakehouse environment. The contribution is not a tool but reproducible evidence and an applicability guide that helps to avoid the up front over specification antipattern.

0.23

STREAM: An Objective-Driven and Uncertainty-Aware Framework for Industrial Energy Data Acquisition

cs.IR ·2026-08-27 ·Zhipeng Ma, Bo Nørregaard Jørgensen, Zheng Grace Ma
publishedunusual similarity 0.62
Abstract · It has been accepted by Energy Informatics.Academy Conference 2026 (EI.A 2026)

Industrial energy management requires datasets that connect energy use with equipment states, production batches, material flows, and process conditions. However, conventional acquisition workflows commonly emphasize connectivity and storage without verifying whether accessible signals satisfy the requirements of a defined energy-performance assessment. This paper presents STREAM, an objective-driven and uncertainty-aware framework comprising Specification of Objectives, Technical Requirements, Resource Mapping, Extraction from Sources, Archival Metadata, and Migration to Database. STREAM is the central workflow: objective-to-data traceability is its end-to-end output, while measurement, temporal, contextual, and processing uncertainty are assessed across all six stages. Compared with the original conceptual STREAM sequence, this paper adds stage-level artifacts, minimum-evidence gates, source-suitability rules, a metadata template, an uncertainty rubric, and case-specific traceability matrices. The framework is validated through two industrial batch-process cases: induction-furnace melting in a foundry and cheese-powder drying using SCADA and production-order data. The results demonstrate that data accessibility is not equivalent to analytical suitability and show how STREAM supports transparent decisions about immediate data use, analytical restrictions, and prioritized infrastructure improvements.

0.36

Value-level data provenance in an electronic laboratory notebook used by humans and language-model agents: An experience report

cs.SE physics.app-ph ·2026-10-20 ·Oleksiy V. Penkov, Boyang Shen
similarity 0.61
Abstract

A published laboratory number can be verified only when its raw file and extraction method are known. This paper presents the value-level provenance architecture of APS ELN, an electronic laboratory notebook used by both researchers and language-model agents. The architecture was motivated by a production audit in a materials laboratory covering projects performed over almost 5 years. Tracing eight baseline studies from sample to stored value showed that the initial ELN structure preserved the link from sample to raw file. Still, the link from raw file to stored value was lost, mainly due to human factors: manual entries carried no file reference or extractor version. To close this gap, the database design and workflow were reworked so that every stored value records its source file, the data file reader/viewer (DFR) plugin and version, any interactive analysis selection, or the server-side formula for derived quantities. An automated drift check re-runs each DFR on the stored files. In two end-to-end validation studies recorded by agents in X-ray reflectivity and tribology, all stored values retained verifiable provenance with zero drift. A comparison with five other platforms shows that none binds individual stored numbers to both their source files and extractor versions.

0.21

CompProv Produces Machine Readable Graphs Encoding Microscopic Algebraic Provenance for Reproducible Computation

cs.SE cs.CE ·2026-10-01 ·Minas Abramyan, Mohammed Alaa Ala'anzy, Nasir Saeed
similarity 0.60
Abstract

The reliability of computational results in scientific research and financial modeling increasingly depends on verifiable traceability, not merely on trust in a reported output. Existing provenance systems operate at the granularity of files, datasets, or pipeline stages, leaving the internal sequence of algebraic transformations connecting an algorithm's inputs to its outputs unrecorded; a rounding error, an undocumented substitution, or a missing intermediate value can propagate to a final result with no recoverable trace. This work presents CompProv, a Java-based, audit-oriented provenance framework that captures lineage at the atomic level of individual algebraic operations by encapsulating numerical values in high-precision wrapper objects, producing a serializable Calculation Provenance Graph (CPG) that persists as a self-contained artifact rather than a discarded byproduct. The framework is evaluated through three heterogeneous case studies: a decentralized-finance NAV calculation, a reconstruction of an interferometric gauge-block calibration in metrology under explicitly documented input assumptions, and a hydrological model performance evaluation. Deterministic replay reproduced each result exactly in a fresh environment, and CPG-based input substitution supported sensitivity analysis without exposing the underlying source code. These results indicate that, provided the CompProv runtime and wrapper classes are available in the replay environment, a CPG allows numerical integrity to be audited without disclosing proprietary business logic, and that its self-contained structure supports temporal auditability once an execution environment has become deprecated. This framework establishes an operation-level foundation for auditable-by-design computational systems, with scaling to high-throughput computing identified as a direction for future work.

0.27

A framework for linking literature-based knowledge integration and infrastructure-supported knowledge integration: Opportunities and challenges from a case study

cs.DL cs.DB ·2026-09-24 ·Mahlet Degefu Awoke, Hadi Ghaemi, Lauren Synder, Markus Stocker, Tilman Brück
survey similarity 0.57
Abstract

Integrating knowledge across disciplines is central to sustainability research, yet most evidence-synthesis methods rely on findings as reported in publications, limiting verification and reuse of underlying data and workflows. We develop a conceptual framework linking literature-based and infrastructure-supported knowledge integration, using a systematic review case study to examine when integration can extend beyond reported findings. We reviewed 37 studies on climate change, violent conflict, and household food security. Literature-based synthesis enabled integration across all included studies, whereas access to reusable outputs was limited: over half provided no data availability statement, 27% reported availability upon request, but reusable data and workflows were available for only 8%. To explore infrastructure-supported integration, we used the TIB Knowledge Loom to represent studies with accessible data and code as machine-readable outputs, and produced a knowledge gap map (KGM) from manually extracted and Loom-derived data, comparing manual and infrastructure-supported synthesis. Where outputs were reusable, synthesis could be produced directly from data and workflows rather than from publications. These findings show that literature-based synthesis can be complemented by infrastructure-supported integration where outputs are accessible and usable, and that advancing knowledge integration depends not only on infrastructures but on making data, code, and workflows accessible, executable, and reusable.

0.16

OntoKG-EQ: Evidence-boundary auditing for analytical knowledge graphs

cs.ET cs.AIcs.DB ·2026-09-08 ·Furqan Nasir, Muhammad Atif Saeed, Muhammad Ehsan, Sher Jeel Ahmad, Abdul Moiz Altaf
similarity 0.57
Abstract · v2: substantially revised and retitled; adds an executable evidence-boundary audit, an NHS public-data application and a release-robustness …

An analytical knowledge-graph result can be supported in its ambient graph while its delivered explanation omits deciding observations. OntoKG-EQ connects competency-question requirements to an executable audit of ambient observations, declared finding support and delivered evidence. A bounded declarative interpreter distinguishes positive, false and unknown numerical status, with same-witness unit readiness reported separately. A public-data application reconstructs England's August 2026 provider-level referral-to-treatment statistical screen. Independent source recomputation reconciles 532 providers with the publisher workbook: 91 meet the inclusive 92% threshold and 441 do not. One additional roster entry remains unavailable rather than zero-filled. All 533 application frames agree with an independently implemented standards-based validation and availability baseline. Transfer requires a disclosed source adapter and arithmetic compilation, but no change to the core interpreter. Twenty-six constructed analytical cases yield 78 status/readiness agreements. Actual financial release replay reproduces two omitted-volume defects: support and delivery suffice for 19 of 21 original findings and all 21 repaired findings after interface adaptation. A separate replay agrees on 487 overlapping financial records. Source audits expose scaling errors and unresolved comparator and temporal assumptions, so financial examples do not establish authenticated market effects. The contribution is a reproducible analytical audit interface and source-reconciled application, not new sufficiency logic or first evidence-boundary isolation. Historical ratings measure preference only; objective human audit benefits and independent production deployment remain untested.

0.40

Integrating Semantics into Research Data Management: Modelling and Validating Materials Science Experiment Workflows

cs.DB ·2026-08-21 ·Samuel García Vázquez, Victor Dudarev, Alfred Ludwig, Markus Stricker, Maribel Acosta
similarity 0.57
Abstract · Preprint of the paper accepted for publication at the International Semantic Web Conference (ISWC) 2026. The final version will be published…

The incorporation of Semantic Web technologies within scientific environments is becoming an increasingly popular Research Data Management (RDM) practice. While ontologies offer flexible, reusable and machine-readable vocabularies to describe domain-specific research data, Knowledge Graphs (KGs) facilitate the integration of heterogeneous data sources into an interoperable collection. Furthermore, KGs offer additional advantages, notably the use of expressive SPARQL queries, or the ability to define complex data validation rules with SHACL. This work describes the modelling of a relational RDM system with an ontology, and the subsequent construction of a KG based on it. Enabled by the highly interconnected nature of research data and experiment workflows present in the system, we not only show how we can easily and reliably build an efficient KG from such domain-specific RDM systems, but also how doing so enables more advanced use cases. This is demonstrated by the modelling of ideal counterparts for the experiment workflows logged in the KG, which are then used to programmatically generate SHACL shapes that fully validate the conformance of the latter. By integrating this functionality within a UI, we allow researchers to plan, reuse, share, and track the progress of their daily experiments.

0.08