CS Paper Radar

Where Does Streaming State Cost Go? A Reproducible Comparison of Flink and Kafka Streams on Kafka

Kiran N Kumar, Santhosh Kumar Saminathan

cs.DC ·submitted 2026-09-23 20:48 UTC · v1
published

Abstract

This paper presents a controlled comparison of exactly-once Kafka pipelines implemented with Apache Flink and Kafka Streams, two engines with different state-management architectures. The state management in Flink occurs through checkpoints in external storage, whereas Kafka Streams restores local state by replaying broker changelogs. The experiments evaluate the effects of the two designs on latency, resource consumption, configuration sensitivity, and recover from injected failures. The engines produced expected outputs across 50 correctness trials, five for each engine-workload combination (exact 95% CI 0.929-1.000). The latencies, however, showed differences when tested in 30 minute trials at a fixed rate of 100 events/sec. The measured ingestion to output interval was 4-6 ms for stateless workloads and 4.3-6.2 s for windowed workloads. The median p99 was 1.97-1.97s for stateless workloads and 5.68-9.92s for windowed respectively. Further, a sensitivity study showed that increasing the durability interval from 1,000 to 10,000 ms shifted the latency distributions in Kafka Streams and Flink between stages. Kafka Streams' W3 median p99 T2-T1 increased from 6,085 to 23,155 ms; Flink's T3-T2 increased from 732 to 8,702 ms. Failure injection experiments demonstrate that latency can interfere with incomplete processing behavior. The results show that exactly-once correctness is a necessary but insufficient measure of stream processing performance. Kafka Streams completed 0/5 stateful JVM-kill and 1/5 in each stateful local-volume-loss trials, while corresponding Flink cells completed 5/5. The cost, placement and magnitude depend on state management, architecture, configuration, and measurement location. Accordingly, evaluations of fault-tolerant streaming systems should holistically measure latency, completion, recovery behavior, and correctness.

Journal ref
Kiran N Kumar, Santhosh Kumar Saminathan, "Where Does Streaming State Cost Go? A Reproducible Comparison of Flink and Kafka Streams on Kafka," International Journal of Computer Trends and Technology (IJCTT), vol. 74, no. 8, pp. 1-8, 2026
DOI
10.14445/22312803/IJCTT-V74I8P101

Closest papers

Nearest neighbours by meaning across the whole index.

HPC-MQBench: Qualification-First Benchmarking on Slurm with a Single-Broker Kafka Evaluation

cs.DC ·2026-10-07 ·Sepehr Mahmoodian, Julian Kunkel
benchmark similarity 0.65
Abstract · 8 pages, 4 figures, 3 tables

Messaging experiments on high-performance computing clusters must coordinate services, clients, and measurement within scheduler allocations. We present HPC-MQBench, a Slurm-orchestrated benchmark that links resumable experiment control to record checks, delivery accounting, and qualification before rate ranking. Its Kafka evaluation placed producers/controller, one broker, consumers, and monitoring on four nodes. With memory-backed logs, 120 workload configurations yielded 99 qualified observations, 14 outside the producer-delivery policy, and seven with invalid evidence. Two validation stages each repeated ten workloads in five blocks. In the final stage, the selected workload qualified in all five observations, with medians of 2,927 mebibytes per second balanced endpoint rate, 1.24 percent pending deliveries, and 2.56 seconds for the 99th-percentile latency. It qualified in only two observations in the earlier stage, where qualification improved after an allocation boundary without establishing a causal allocation effect. Selection is conditional on the stage and delivery policy. Resource measurements did not isolate a unique bottleneck. The contribution is a framework for distributed experiment control and auditable configuration selection at a fixed broker count; multi-broker support and scaling experiments remain future work.

0.18

Gateway-Free Kafka Streaming from FPGA-Based Detectors: A Bare-Metal Pipelined Producer on Zynq-7000

cs.NI cs.AR ·2026-11-17 ·Fernanda Zapata Bascuñán, Matías Hampel
published similarity 0.62
Abstract · Accepted for pucblishing at IEEE Explore proceedings of ARGENCON 2026

We present a bare-metal Apache Kafka producer implemented on a Zynq-7000 System-on-Chip (SoC) for event-driven data acquisition. Events are generated in Programmable Logic (PL), transferred to the Processing System (PS) through AXI4-Stream and AXI DMA, and published to a standard Kafka broker without an intermediate gateway computer. The PS constructs ProduceRequest messages and transmits them through the lwIP raw TCP/IP API. A configurable pipeline limits the number of outstanding Kafka requests. Pipeline depths of D = 1, 2, 4, and 8 were evaluated with two independent DMA sources. For each depth, 20 independent runs of 5,000 events were acquired, for 400,000 event records in total. Each event carries a 32-bit frame counter and a 32-bit PL timestamp. PS-side instrumentation records DMA start and completion, publication, successful local tcp_write() acceptance, and reception of the corresponding Kafka acknowledgement. Measurement records are exported to a separate Kafka topic after the timed event stream has completed. Each of the 80 runs contained 5,000 records with frame values from 0 to 4,999, with no sequence discontinuities in the exported metric records and no recorded Kafka errors. Mean operational throughput was approximately 5.3 kevents/s at D = 1, 9.4-9.6 kevents/s at D = 2, 16.2 kevents/s at D = 4, and 16.8-17.5 kevents/s at D = 8. The median publish-to-acknowledgement interval was 1.62-1.64 ms at D = 1, 0.98 ms at D = 2, 0.66 ms at D = 4, and 0.83-0.86 ms at D = 8. At D = 4, the corresponding P95 values were 0.81-0.83 ms.

0.35

CLASP: Chained-Request-Aware Scaling and Operator Placement for Serverless Stream Processing

cs.DC ·2026-08-29 ·Tianyu Qi, Maria A. Rodriguez, Rajkumar Buyya
similarity 0.56
Abstract

Stateful serverless (Function-as-a-Service) environments, whose workers host state servers, are increasingly used for stream processing. A stream application is a pipeline of operators, where each operator forwards intermediate data downstream through a chained request. As input rates fluctuate, the system should adjust operator parallelism and place instances across workers to sustain the incoming rate. Existing approaches do so without fully accounting for chained-request overhead, leading them to misestimate the required number of workers. Too few leave the cluster unable to keep up with the input rate, while too many route a larger fraction of chained requests across worker boundaries, increasing end-to-end latency. We propose CLASP, a scaling and scheduling strategy for stream processing in stateful serverless environments. At runtime, CLASP estimates execution cost and chained-request cost from observed metrics. Under a capacity model that covers the two costs, it adjusts operator parallelism and packs operators onto the fewest workers that can sustain the target input rate. Once a scaling decision is made, CLASP migrates each operator's state together with its instances, thereby minimizing execution pause time. Experiments show that CLASP improves throughput by up to 3.3x and reduces median end-to-end latency by up to 76% compared with state-of-the-art scaling strategies.

0.11

From Isolated Protocols to Integrated Pipelines: Benchmarking IoT Communication in the MASA Living Lab

cs.NI ·2026-11-11 ·Antonio Solida, Enrico Rossini, Marcello Pietri, Marco Picone, Carlo Augusto Grazia
publishedbenchmark similarity 0.54
Abstract · Accepted for publication at the IEEE 2027 Consumer Communications and Networking Conference; track T3: Testbeds, Experimentation and Dataset…

The performance of real-time IoT communication systems depends not only on individual transport protocols, serialization formats, and data-processing stages, but also on how these components interact within the complete communication pipeline. Selecting an appropriate communication stack is therefore application-dependent and requires considering processing capabilities, network characteristics, communication patterns, and reliability requirements. This paper provides an experimental, evidence-based analysis of these interactions through a controlled benchmark of integrated communication pipelines deployed in the MASA living lab. We jointly evaluate local processing overhead, end-to-end latency, jitter, throughput, and delivery reliability across different combinations of transport, messaging, serialization, and compression mechanisms. Results show that component-level processing costs do not directly predict end-to-end performance. In particular, differences of only tens of microseconds in serialization or compression costs can coexist with several-millisecond differences in end-to-end latency, while middleware buffering and transport behavior substantially affect timing performance. The results demonstrate that communication-stack components should be evaluated as integrated pipelines and under representative operating loads rather than ranked independently.

0.36

A Controlled Experimental Evaluation of Configuration and Cache Architecture in LLM Serving Engines

cs.DC cs.LGcs.PF ·2026-11-04 ·Muhammad Faizan Anwar, Sina Keshvadi
benchmarkcode/data similarity 0.54
Abstract · 29 pages, 14 figures, 7 tables. Artifact: https://github.com/FaizanAnwar01/llm-serving-config-eval

Multi-agent LLM applications repeat most of their prompt tokens across requests, and serving engines reuse the cached state of shared prefixes. vLLM and SGLang reuse it at different granularities, and benchmarks of their default deployments report large performance gaps, usually credited to the cache. Such comparisons rarely control for other configuration differences. One of them, whether a long prompt is prefilled in chunks, is set by each engine's defaults from the hardware and the release, and it confounds the comparison. We built a reproducible testbed on Kubernetes and a seeded multi-agent workload generator, and compared the two engines under vendor defaults and with chunked prefill matched. SGLang's configuration is the same in both arms, which makes it an internal control across independently provisioned deployments. Under defaults, vLLM's median time to first token is up to 64x SGLang's. Matching one setting removes 78% of the capacity gap where that gap is largest and 30 to 67% of the latency gap. Near saturation the remaining latency gap reflects queue growth. A fluid queue model driven only by a 4.4% throughput shortfall reproduces it, and shows that such a gap scales with the length of the benchmark run. What survives matching is a tail-latency advantage of 2.4 to 6.4x at equal throughput, consistent with cache-aware ordering of waiting requests. We derive six recommendations for comparing serving engines.

0.32

CheckCo: Checkpoint-Coordinated Load Balancingfor Skewed Event Streams at Scale

cs.DC ·2026-11-17 ·Muhammad Anis Uddin Nasir, Reza Sadraei
similarity 0.54
Abstract · This paper is submitted to ICDE 2027 Industry and Application Track

Event streams are rarely uniform: a few dominant keys can generate orders of magnitude more events than the rest, overloading the workers that receive them and capping throughput. We present CheckCo, a checkpoint-coordinated load-balancing algorithm built on a dynamic oracle that estimates per-key frequencies from the live stream, with no prior statistics. Workers count events per key locally; at each checkpoint barrier, the counts are merged into a partitioning function that divides each key into a number of splits roughly proportional to its estimated frequency and packs the splits onto workers, and this function routes the next checkpoint window. We derive approximation bounds on the resulting load imbalance from classical scheduling results and bound its total key fan-out. On a production pipeline processing over 100 billion events per day, the most loaded worker receives only 9.8% more events than the average over a day (imbalance ratio 1.10), against 8.03 for key-based routing in simulation; in simulation, CheckCo also needs less than half the fan-out of earlier partitioners that reach comparable balance. Shifts in the key distribution are reflected in routing from the next checkpoint on, and the load stays balanced through job restarts.

0.21

DeltaHarness: Self-Evolving Harnesses for Resource-Adaptive Tool Execution

cs.LG cs.PF ·2026-11-16 ·Lecheng Yan, Jianze Lin, Xicheng Han, Xiantao Xu, Wenxi Li +1 more
similarity 0.53
Abstract · preprint

Tool-using agents rely on execution harnesses to organize and schedule tool operations. The same task and resource description can conceal runtime conditions that reverse the latency ordering of valid execution plans. Resolving this uncertainty requires firsthand execution in the target setting. We introduce DeltaHarness, an execution-guided policy-search method that adapts agent harnesses from cold-start measurements. Its QHI loop uses the observed response to update runtime dispatch, revise physical plans, and select experiments for later proposals. An independent confirmation panel controls policy replacement. With fixed dispatch, a revised materialization policy reduces complete-delivery latency by 30.0% on one video-editing task across three fresh pairs. On held-out tasks, the evolved policies reduce latency by 8.2% relative to the best fixed plan in one seed and by 4.6% relative to the initial proposal in another.

0.26

Koala: Non-disruptive Reconfiguration for Long-running Stateful Dataflows

cs.DC ·2026-10-20 ·Lei Huang, Sakshi Sharma, Chenxu Yu, Zikun Wang, Yuanli Wang +3 more
similarity 0.53
Abstract

Stateful dataflow systems must decide where state lives relative to the tasks that process it. For long-running applications that require frequent reconfiguration due to changes in resource allocation, partitioning, or task placement, this design choice presents a core trade-off between adaptability and performance. Co-locating state with computation ensures low latency at steady state but makes reconfiguration disruptive, as any change forces state migration. On the other hand, keeping state in an external store makes reconfiguration seamless, but incurs overhead on every access. We present Koala, a reconfiguration protocol for distributed dataflow systems that retains the operational performance of embedded-state architectures while matching the disruption-free adaptability of external-state designs. Unlike prior protocols which relocate all state regardless of whether it will soon be accessed, Koala migrates state in a fine-grained, lazy manner. To achieve that, it employs a lightweight ownership transfer mechanism that safely decouples event routing from state location. We implement Koala in a new streaming dataflow system and evaluate it on real-world and synthetic benchmarks. When compared with the state-of-the-art protocol DRRS, Koala consumes backlog up to 23x faster and transfers 8.7x less state on average during reconfiguration.

0.21