CS Paper Radar

Gateway-Free Kafka Streaming from FPGA-Based Detectors: A Bare-Metal Pipelined Producer on Zynq-7000

Fernanda Zapata Bascuñán, Matías Hampel

cs.NI cs.AR ·submitted 2026-11-17 12:25 UTC · v1
published

Abstract

We present a bare-metal Apache Kafka producer implemented on a Zynq-7000 System-on-Chip (SoC) for event-driven data acquisition. Events are generated in Programmable Logic (PL), transferred to the Processing System (PS) through AXI4-Stream and AXI DMA, and published to a standard Kafka broker without an intermediate gateway computer. The PS constructs ProduceRequest messages and transmits them through the lwIP raw TCP/IP API. A configurable pipeline limits the number of outstanding Kafka requests. Pipeline depths of D = 1, 2, 4, and 8 were evaluated with two independent DMA sources. For each depth, 20 independent runs of 5,000 events were acquired, for 400,000 event records in total. Each event carries a 32-bit frame counter and a 32-bit PL timestamp. PS-side instrumentation records DMA start and completion, publication, successful local tcp_write() acceptance, and reception of the corresponding Kafka acknowledgement. Measurement records are exported to a separate Kafka topic after the timed event stream has completed. Each of the 80 runs contained 5,000 records with frame values from 0 to 4,999, with no sequence discontinuities in the exported metric records and no recorded Kafka errors. Mean operational throughput was approximately 5.3 kevents/s at D = 1, 9.4-9.6 kevents/s at D = 2, 16.2 kevents/s at D = 4, and 16.8-17.5 kevents/s at D = 8. The median publish-to-acknowledgement interval was 1.62-1.64 ms at D = 1, 0.98 ms at D = 2, 0.66 ms at D = 4, and 0.83-0.86 ms at D = 8. At D = 4, the corresponding P95 values were 0.81-0.83 ms.

Comment
Accepted for pucblishing at IEEE Explore proceedings of ARGENCON 2026

Closest papers

Nearest neighbours by meaning across the whole index.

Where Does Streaming State Cost Go? A Reproducible Comparison of Flink and Kafka Streams on Kafka

cs.DC ·2026-09-23 ·Kiran N Kumar, Santhosh Kumar Saminathan
published similarity 0.62
Abstract

This paper presents a controlled comparison of exactly-once Kafka pipelines implemented with Apache Flink and Kafka Streams, two engines with different state-management architectures. The state management in Flink occurs through checkpoints in external storage, whereas Kafka Streams restores local state by replaying broker changelogs. The experiments evaluate the effects of the two designs on latency, resource consumption, configuration sensitivity, and recover from injected failures. The engines produced expected outputs across 50 correctness trials, five for each engine-workload combination (exact 95% CI 0.929-1.000). The latencies, however, showed differences when tested in 30 minute trials at a fixed rate of 100 events/sec. The measured ingestion to output interval was 4-6 ms for stateless workloads and 4.3-6.2 s for windowed workloads. The median p99 was 1.97-1.97s for stateless workloads and 5.68-9.92s for windowed respectively. Further, a sensitivity study showed that increasing the durability interval from 1,000 to 10,000 ms shifted the latency distributions in Kafka Streams and Flink between stages. Kafka Streams' W3 median p99 T2-T1 increased from 6,085 to 23,155 ms; Flink's T3-T2 increased from 732 to 8,702 ms. Failure injection experiments demonstrate that latency can interfere with incomplete processing behavior. The results show that exactly-once correctness is a necessary but insufficient measure of stream processing performance. Kafka Streams completed 0/5 stateful JVM-kill and 1/5 in each stateful local-volume-loss trials, while corresponding Flink cells completed 5/5. The cost, placement and magnitude depend on state management, architecture, configuration, and measurement location. Accordingly, evaluations of fault-tolerant streaming systems should holistically measure latency, completion, recovery behavior, and correctness.

0.62

A Kafka-Centric Communication Fabric for Near-Real-Time, Cloud-Replicated Closed-Loop Manufacturing Process Control

cs.NI ·2026-09-24 ·Zhengyang, Gu, John Burtenshaw, Joseph E. Hernandez, Chris Couch
published similarity 0.61
Abstract · Accepted to the IEEE Real-Time Communications Conference and Expo 2026

Smart manufacturing needs to move sensor data off the plant floor, react to it, and feed decisions back to actuators within bounded time. Programmable logic controllers (PLCs) handle fast, deterministic, safety-critical actuation, but they are not designed for the higher-level functions required by Industry 4.0, such as predictive maintenance, machine learning inference, and cross-facility analytics. These functions need a scalable, durable, and observable communication substrate. We present the communication architecture of a production system that provides this substrate and closes the loop back to the plant in near real time. The design is built from industry-standard components: Apache Kafka as the streaming backbone, OPC-UA for PLC connectivity, a relational time-series database for persistence, and JSON for serialization. The novelty is architectural. We show how these standards are integrated for closed-loop industrial control through four design choices: a single event stream per production line serves control, monitoring, machine learning, and durable recording, allowing one producer to serve many independent consumers; a protocol bridge converts polled OPC-UA traffic into publish/subscribe streams, aligns per-signal timestamps to a common time base to remove cross-signal jitter, and provides a symmetric actuation path; a transport technique carries sub-second process dynamics at a coarser publication cadence by packing timestamped samples into fixed-order arrays; and an edge-to-cloud replication scheme keeps the edge authoritative, so local control continues during wide-area network outages while cloud analytics operate on replicated data. We describe the loop latency budget, report measured broker transport latency, and discuss operational experience. The system provides soft, near-real-time behavior rather than hard real-time guarantees.

0.36

FIREQ: FPGA Instrumentation for Readout and Qubit control

eess.SY quant-ph ·2026-08-29 ·Giuseppe La Capra, Fabio Calabrese, Giorgio D'Amico, Christian Conti, Andrea De Simone +6 more
code/data similarity 0.59
Abstract · 15 pages, 9 figures

We present FIREQ (FPGA Instrumentation for Readout and Qubit control), an open-source RFSoC-based framework for the control and readout of superconducting qubits. FIREQ combines a modular AXI-compliant firmware architecture with a PYNQ-based software stack designed to support extensible hardware integration, deterministic experiment timing, and low-overhead execution of repeated calibration and characterization workflows. The firmware implements direct RF synthesis and acquisition, trigger-based sequencing, programmable pulse generation, frequency-multiplexed readout, and memory-efficient acquisition and waveform buffering. The software adopts a client-server architecture with streamed data transfer and dependency-aware configuration updates to reduce host-device and reconfiguration overhead during parameter sweeps. On an AMD Zynq UltraScale+ RFSoC ZCU216, FIREQ generates RF pulses up to 9.3 GHz with a pulse-duration resolution of 107 ps and an event-timing resolution of 1.7 ns. FPGA resource utilization is compared with representative open-source RFSoC control frameworks, showing a low BRAM footprint while retaining full-rate I/Q generation and acquisition. The RF output is characterized in terms of phase noise, noise spectral density, and inter-channel timing skew. End-to-end operation is validated on a superconducting qubit through resonator spectroscopy, Rabi, Ramsey, and relaxation measurements, yielding T1 = 6.94 us and T2* = 13.50 us. FIREQ can therefore be used both as a qubit-control platform and as an experimental environment for evaluating alternative control and readout IP architectures.

0.35

From Isolated Protocols to Integrated Pipelines: Benchmarking IoT Communication in the MASA Living Lab

cs.NI ·2026-11-11 ·Antonio Solida, Enrico Rossini, Marcello Pietri, Marco Picone, Carlo Augusto Grazia
publishedbenchmark similarity 0.57
Abstract · Accepted for publication at the IEEE 2027 Consumer Communications and Networking Conference; track T3: Testbeds, Experimentation and Dataset…

The performance of real-time IoT communication systems depends not only on individual transport protocols, serialization formats, and data-processing stages, but also on how these components interact within the complete communication pipeline. Selecting an appropriate communication stack is therefore application-dependent and requires considering processing capabilities, network characteristics, communication patterns, and reliability requirements. This paper provides an experimental, evidence-based analysis of these interactions through a controlled benchmark of integrated communication pipelines deployed in the MASA living lab. We jointly evaluate local processing overhead, end-to-end latency, jitter, throughput, and delivery reliability across different combinations of transport, messaging, serialization, and compression mechanisms. Results show that component-level processing costs do not directly predict end-to-end performance. In particular, differences of only tens of microseconds in serialization or compression costs can coexist with several-millisecond differences in end-to-end latency, while middleware buffering and transport behavior substantially affect timing performance. The results demonstrate that communication-stack components should be evaluated as integrated pipelines and under representative operating loads rather than ranked independently.

0.36

High Speed Network Processing for Scientific Computing in the Open Cloud Testbed

cs.NI cs.DC ·2026-10-05 ·Miriam Leeser, Michael Zink, Suranga Handagala, Carlos Ruben Dell'Aquila, Verena Martinez Outschoorn +1 more
published similarity 0.57
Abstract · Paper accepted to the Twelfth International Workshop on Heterogeneous High-performance Reconfigurable Computing (H2RC'26) Held in conjunctio…

This paper presents an FPGA-based SmartNIC architecture designed to address critical network challenges in data-intensive scientific computing. As scientific workflows increasingly demand high-bandwidth data movement and real-time monitoring across wide-area networks, the Open Cloud Testbed (OCT) provides an ideal platform for investigating in-network processing solutions at line rate. Our approach focuses on implementing network visibility, optimization, and orchestration capabilities through FPGA acceleration. The proposed system leverages P4-programmable data-plane functions to support packet parsing, marking, telemetry extraction, traffic steering, and machine learning inference, all without requiring host CPU intervention or modifications to application servers. As a concrete example, our framework integrates the SciTags standard, which enables associating network traffic with scientific context through packet marking. We implement a high-performance network bridge that counts packets by their SciTag values, as identified in the IPv6 Flow Label. Experimental evaluation on AMD U280 FPGAs in OCT demonstrates the feasibility of this approach, showing imperceptible processing latency and line-rate packet handling, while accurately monitoring scientific data flows.

0.22

RyBench: Runtime Verification Benchmarking Framework over Publish/Subscribe Networks

cs.LO cs.PFcs.SE ·2026-11-17 ·Mehmet Hilmi Dündar, Doğan Ulus
publishedbenchmark similarity 0.57
Abstract · In Proceedings FMAS 2026, arXiv:2611.21375

Autonomous systems built as distributed, node-based systems communicate via publish/subscribe middleware. Runtime Verification (RV) tools are used to monitor those systems against formal specifications, receiving monitored events through the pub/sub channel. An RV tool's practical value depends on its monitoring throughput capacity and verdict latency, besides its correctness, so a systematic benchmarking setup is needed to evaluate how efficiently these tools run. This paper presents RyBench, a pub/sub-based benchmarking framework that acts as a synthetic event publisher in place of a monitored system, providing a common benchmarking setup. RyBench separates benchmark generation, serialization, and transport into distinct components, enabling the combination of different event-generation strategies, data formats, and transport layers. We also evaluate the framework's throughput capacity across these configurations using Open Simulation Interface (OSI) schemas and Timescales properties, and illustrate the end-to-end benchmarking flow through an application to Reelay and an integration with MonPoly.

0.42

LatencyLab: A DPDK-Based P4 Pipeline Latency Measurement Framework for FPGA SmartNICs

cs.DC ·2026-09-30 ·Pavani Kuppili, Zhaoyang Han, Yicheng Qian, Suranga Handagala, Michael Zink +2 more
published similarity 0.57
Abstract · This paper appeared in the 30th Annual IEEE High Performance Extreme Computing (HPEC) https://ieee-hpec.org/ where it won an outstanding pap…

P4-programmable FPGA SmartNICs place packet processing directly on the wire, but open FPGA P4 toolflows do not expose timestamping at the pipeline boundary, so the latency a P4 program adds on the target FPGA is rarely measured. This paper presents LatencyLab, a DPDK-based measurement framework for FPGA P4 pipeline latency that needs neither PHC/PTP support on the datapath nor clock synchronization. The FPGA's two ports share a network segment, so the switch multicasts a copy of each probe packet to both: one copy passes through the VitisNetP4 pipeline, the other through a matched bypass path. A kernel-bypass DPDK receiver busy-polls both ports and timestamps every packet with the CPU timestamp counter (TSC) as it is retrieved from the NIC's receive circular buffer. The arrival-time difference of the two copies isolates the pipeline latency after calibration against a null bitstream carrying the same traffic; transmit time cancel in the subtraction. We evaluate four VitisNetP4 programs on an AMD Alveo U280, probing each with a 20,000-packet trace measured ten times per session over five independent sessions, all TSC-timestamped and reflected for hardware timestamping. The measured latency distributions are tight and reproducible: 99% of packets fall within 20 ns of the median, session medians repeating within 1 to 2 ns (FiveTuple 107/137 ns, Forward 149 ns, RemoveHeader 177 ns, Checksum 364 ns at 250 MHz). Two independent checks agree with the framework: a kernel-free reflector returns every probe pair to a ConnectX-5 NIC whose adapter clock reproduces the measured distributions within a few nanoseconds, quantile by quantile, and every measured packet falls 18 to 22 clock cycles below the vendor's worst-case latency bound.

0.42

An Open-Source, Event-Driven Pipeline for Cryptocurrency Market Data: Ingestion, Forecasting, and On-Chain Fraud Detection

cs.AI cs.LG ·2026-08-30 ·Basil Sajid Shaikh, Melrick Mascarenhas, Nuzhat Faiz Shaikh
benchmarkcode/data similarity 0.57
Abstract

Cryptocurrency markets generate high-frequency, multi-source data that is expensive to work with unless a team already has commercial-grade streaming and warehousing infrastructure in place. This paper describes a fully open-source pipeline that reproduces the behavior of a cloud-native, event-driven system -- file arrival triggering a message, a message triggering compute -- entirely on commodity hardware, using Apache Kafka and a filesystem-watching poller in place of managed cloud triggers. The pipeline partitions historical Gemini exchange data into hourly and minutely files, ingests them asynchronously through two independently grouped Kafka consumers (one for audit logging, one for Spark-triggered ETL), and lands cleaned output in a PostgreSQL warehouse with historical and aggregated schemas plus asset-specific data marts. We use the resulting Bitcoin data mart to compare a seasonal ARIMA model against a single-layer LSTM network for price forecasting, and separately apply Random Forest and Gradient Boosting classifiers, with additional engineered features, to the public Ethereum fraud detection benchmark introduced by Farrugia et al. We report the architecture, the modeling methodology, and the resulting metrics, and we are explicit about the limitations of comparing forecasts issued at different horizons and of evaluating fraud detection on a static, already-labeled dataset.

0.31