CS Paper Radar

Enabling Reuse for Data-Sharing Pipelines in Federated Environments

Sepideh Masoudi, Maria C. Borges, Eduardo Brito, Sebastian Werner, Stefan Tai

cs.DC ·submitted 2026-08-19 10:00 UTC · v1
top venueposition

Abstract

Data mesh architectures enable decentralized data sharing through domain-owned data products, but supporting diverse consumers in federated settings often requires customized data-sharing pipelines. As the number of consumers grows, this leads to a proliferation of pipelines, increasing design and maintenance complexity. We observe that such pipelines frequently exhibit substantial structural overlap. In this paper, we argue that reuse should serve as a guiding principle to address this challenge. We define reuse in data-sharing pipelines as the systematic use of existing data assets and transformation logic across pipelines, and identify reuse opportunities at both design time and runtime. We analyze the associated challenges and outline a reuse-oriented design approach, supported by a reference architecture. A preliminary evaluation demonstrates the potential of reuse to reduce redundancy and improve manageability, providing a pathway toward more scalable and sustainable federated data sharing.

Closest papers

Nearest neighbours by meaning across the whole index.

Beyond Service Discovery: Coverage Awareness in LLM-Driven Service Composition

cs.DC ·2026-11-15 ·Sepideh Masoudi, Ronja Froelian, Stefan Tai
similarity 0.69
Abstract

Large language models increasingly support the automated discovery and composition of services. In federated data ecosystems, this creates new opportunities for reusing existing data products and transformation services when constructing service-based data exchange modeled as data-sharing pipelines. However, finding relevant reusable assets does not imply that the available assets are sufficient to satisfy a consumer request. In this paper, we distinguish reuse discovery, the ability to identify relevant published assets, from reuse coverage, the extent to which the available assets support the transformations required by a request. We further consider coverage awareness, the ability of a composition system to recognize insufficient coverage. We investigate this distinction using PipeFinder, an LLM-driven framework for composing data-sharing pipelines, and evaluate 99 consumer requests against a fixed catalogue of data products and transformation services. Our experiments show that relevant data products can be discovered reliably, while successful composition remains considerably more difficult. More importantly, when required transformation services are missing, the framework typically constructs a pipeline from the available services rather than recognizing that the request cannot be satisfied. Explicitly incorporating coverage checking into the planning procedure substantially improves recognition of such cases. These results suggest that coverage awareness should be treated as an explicit concern in LLM-driven service composition and cannot be assumed to follow from improved discovery or model capability.

0.18

ASAP: Reimagining the Data Lifecycle using Application Semantic-Aware Processing

cs.DB cs.NIcs.PF ·2026-08-13 ·Milind Srivastava, Zeying Zhu, Yajie Zhou, Yancheng Yuan, Fenghao Dong +3 more
position similarity 0.59
Abstract · 7 pages, 3 figures

Across many domains (e.g., observability, networking, security), data processing pipelines face what we refer to as the CSP problem: achieving low Cost at large Scale, while maintaining high Performance. In response, we see several efforts to tackle CSP in various stages of the Collect-Transmit-Store-Analyze data lifecycle; such as approximate query processing in databases or sketches in network routers. Our work is driven by the simple insight: "seeing the forest for the trees". These proposed solutions (e.g., AQP, sketching, compression, rollups) addressing CSP share a common property - they exploit semantic-preserving opportunities to support application needs. In this paper, we make a case for ASAP, a paradigm that makes Application Semantic-Aware Processing (ASAP) a first-class design principle in data processing pipelines. We argue that by taking a unified view across ASAP primitives developed in different domains, across the entire data lifecycle, we can unlock new opportunities to tackle the CSP problem. In particular, we can: (i) enable novel cross-lifecycle optimizations such as analytics run directly on sketches computed at the source; (ii) leverage primitives developed in other application domains; and (iii) enable widespread adoption of these powerful techniques. We discuss research challenges in socializing the benefits of the ASAP paradigm, and show preliminary evidence that adopting ASAP can yield up to 3 orders of magnitude improvements in the CSP tradeoff for many application domains.

0.07

DS2-Based Cross-Data-Space Interoperability for Precision Agriculture

cs.DC ·2026-09-15 ·Katerina Kyriakou, Ilias Syrigos, Ioannis Moutsinas, Panagiotis Tzimotoudis, Thanasis Korakis
published similarity 0.58
Abstract · Accepted at the 9th Conference on Cloud and Internet of Things (CIoT 2026)

Despite the strategies of modern precision agriculture to leverage the integration of legacy agricultural systems, the challenges of IoT data fragmentation, farmers' sovereignty preservation, and limited interoperability still persist. This paper presents our work, conducted within the Horizon Europe DS2 project (DataSpace, DataShare 2.0), that applies an interoperability-oriented framework supporting participants of different agricultural data spaces to share data products and services under secure, sovereign, and transparent methods. The suggested methodology follows a layered reference architecture, where each layer consists of independent operational modules that facilitate the inter-sector data exchange between DigiAgro and AgroScience Data Spaces. The result of this work is an automated ecosystem for sharing diverse farm IoT measurements, satellite images and metrics, weather forecasts, and analytics services across distinct data spaces, aiming to generate accurate recommendations on crop practices, such as irrigation schedules, that farmers and agronomists will rely on to increase crop production while maintaining sustainability.

0.32

Trustless Accountable Data Sharing for Supply Chains: A Reference Architecture and MoreMedDiet Proof-of-Concept

cs.ET ·2026-08-31 ·Michal Kit, Montassar Naghmouchi, Maryline Laurent, Badis Hammi, Hella Kaffel Ben Ayed +4 more
similarity 0.57
Abstract

Modern supply chains are increasingly dependent on the exchange of sensitive business data between heterogeneous stakeholders who do not trust each other. %under partial trust. Conventional integration approaches typically rely on centralized intermediaries or require extensive data disclosure, which hinders collaboration, compliance, and innovation. We propose a generic reference architecture for privacy preserving, trustless data sharing in supply chains. The architecture combines permissioned distributed ledgers, IoT-based evidence ingestion, and modular privacy mechanisms to enable controlled transparency: stakeholders can extract verifiable insights and support audits while minimizing data exposure. We instantiate the architecture in the context of the MoreMedDiet project, which promotes sustainable farm-to-fork food systems and adoption of the Mediterranean diet. By mapping the MoreMedDiet use case into our architecture, we demonstrate how channel topologies, IoT-to-ledger data flows, and smart contract-based quality assurance (QA) can support secure data exchange, transparency, interoperability, and regulatory alignment. Our proof-of-concept illustrates how Self-Sovereign Identity (SSI) and W3C Verifiable Credentials (VCs) can provide portable and auditable access governance over Hyperledger Fabric channels. The resulting design offers a reusable blueprint for multi-actor supply chains, including agri-food and pharmaceutical domains. Moreover, the performance achieved -- presenting a verified credential to the next stakeholder in the supply chain in 130 ms -- is promising and demonstrates the potential applicability of the approach in real-world supply chain environments.

0.11

Automated consolidation of dispersed DSO data resources into a consistent digital twin of the distribution grid

eess.SY ·2026-11-03 ·Fabian Böhm, Benjamin Sawicki, Claudio Stucki, Pulkit Nahata, Gabriela Hug +1 more
published similarity 0.57
Abstract · This paper is a preprint of a paper accepted by IET conference proceedings and is subject to Institution of Engineering and Technology Copyr…

The energy transition causes a growing number of distributed flexible energy resources in the low voltage grid. To utilize that flexibility, automation and control algorithms are needed, which typically are developed in theoretical research. We build a cyberphysical algorithm test bed to bridge the gap between theoretical research and practical application in cooperation with a local DSO in Walenstadt, Switzerland. To obtain suitable monitoring and data analysis capabilities from their existing data landscape, we propose tool-independent data harmonization measures in preparation for a digital twin of the grid. For data extraction from high-security network zones we developed a secure pipeline that implements a logical data diode ensuring one-way traffic only. Furthermore, we established a test community of voluntary prosumer households, whose flexible assets were made accessible through a common backend, giving us actuation capabilities. The presented publish-subscribe architecture is not only suitable to integrate various data sources into a central data collection, but also to manage access privileges of different stakeholders. This could support unbundling of an organization's data flows and the implementation of data space concepts.

0.48

Live Adaptation of Dataflow Stream Processing for Hybrid Edge-Cloud Analytics

cs.DC ·2026-10-26 ·Yuanli Wang, Prateek Jain, Lei Huang, Le Xu, Vasiliki Kalavri
similarity 0.56
Abstract

Dataflow streaming systems, like Apache Flink and Spark Streaming, are integral to modern business analytics, providing real-time insights across industries. However, for emerging hybrid edge-cloud analytics pipelines, neither processing everything in the cloud nor static edge offloading can reliably sustain the target rate under dynamic WAN conditions and limited edge resources. In this paper, we explore how these datacenter-centric platforms can be extended to support such hybrid analytics pipelines. Our goal is to maintain full transparency for user applications, while retaining built-in fault tolerance, out-of-order processing, and existing ecosystem integrations. Achieving this is challenging because existing dataflow engines assume homogeneous resources, rely on static task placement, and do not support adaptation under dynamic network and edge conditions. We introduce Sluice, an adaptive data stream processing system for hybrid clouds, compatible with dataflow streaming engines. Sluice's key contributions are: (i) an automatic query rewriter that decomposes applications into independent, fault-tolerant segments, and (ii) a live routing mechanism that shifts computations between edge and cloud without pausing execution while preserving event-time semantics. We implement Sluice on Apache Flink and evaluate it on a physical testbed with Raspberry Pi devices, as well as on a large simulated testbed of 100 VMs. Across five real-world applications, our results show that Sluice seamlessly shifts computations between edge and cloud, while consistently achieving higher throughput than baseline systems.

0.08

Multi-center Medical Data Mining with FL-Net - A One-stop Shop for Federated Learning

cs.LG cs.CRcs.DC ·2026-09-17 ·Simon Süwer, Julian Klemm, Elisa Acitelli, Mathieu Almeida, Lucia Altucci +37 more
similarity 0.56
Abstract · 69 pages, 8 figures, includes supplementary material

Federated learning enables collaborative training without sharing patient-level data, but most studies remain simulations. Based on five requirements derived from the literature, we analyzed 14 FL frameworks and found that none fully satisfied these requirements. We present FL-Net, a novel federated clinical research framework to fulfill all requirements. It integrates modular data harmonization, data discovery, disclosure control, securely built versioned FL-Net-Tools and containerized federated workflow execution into a persistent network. It enables the re-use of harmonized data and workflows across studies. FL-Net's end-to-end capabilities were evaluated through harmonization, cross-study patient discovery across MIMIC and US-130, and reproducible, audited federated workflows with up to 50 concurrent clients. FL-Net is being developed within the dAIbetes and Microb-AI-ome EU projects and will cover over 800,000 patients across 10 hospitals in 9 countries covering longitudinal and single point in time data, FL-Net provides a practical foundation for interoperable, reproducible, and privacy-preserving multicenter clinical research.

0.24

Chronos: Efficient Bolt-on Branching Across Data Stores for Stateful Agentic Applications

cs.DB ·2026-09-14 ·Xinjing Zhou, Jason Mohoney, Samuel Madden, Michael Stonebraker, Lei Cao
similarity 0.56
Abstract

Data-centric applications increasingly use speculative execution to explore multiple candidate paths where each path modifies state distributed across heterogeneous data stores. This trend is intensified by the rise of tool-calling agents. Hence, applications need data systems that can create branches quickly, isolate state-modifying paths, and merge changes consistently across stores without imposing substantial query overhead. Existing systems provide only partial support, forcing applications to coordinate branches and merges manually, which increases overhead and risks inconsistent cross-store state. To solve this problem, we introduce Chronos, a bolt-on system that provides branching capability across heterogeneous data stores. We make two contributions. First, Chronos introduces a compact interval-based versioning technique that enables efficient branching and data sharing through simple query rewrite. Second, Chronos introduces a bolt-on architecture that separates branch management from data path within each store. Combined with interval-based versioning, this separation provides atomic cross-store visibility for merges and enables Chronos to support diverse data stores without modifying their engines. We implement Chronos for PostgreSQL, SQLite, DuckDB, Qdrant, and a DBMS-backed filesystem. We evaluate it using cross-store agent workflows, MCTS-style exploration, and per-store benchmarks. Chronos runs MCTS-style exploration up to 16.7x faster than existing approaches while maintaining practical query performance across the underlying stores. Under concurrent cross-store workflows, Chronos prevents partially visible merges while substantially outperforming serialized execution.

0.06