CS Paper Radar

Packets, Transactions and Queues: Design Principles for HFT Systems from a Measurement Study of CME Market Data

Vincent Maciejewski

cs.DC q-fin.TRcs.PFq-fin.ST ·submitted 2026-09-26 18:22 UTC · v2
code/data

Abstract

HFT systems are conventionally built as a single-threaded event loop, on the rule that every thread hop adds latency. We test that rule against a measurement study of more than a year of CME market data for the NQ front-month contract, following every packet and matching-engine transaction through the feed's two exchange timestamps, and checking the results against a live production receiver. Packets arrive in near-critical self-exciting clusters that belong to the matching engine's transactions, not to how the exchange packs them. The engine often processes consecutive transactions within a fraction of a microsecond, while the market-data publisher sends at most one packet per publisher period of about 7.5 microseconds, so a burst reaches the receiver as a train of packets one period apart. This yields design principles for HFT systems. First, a receiver that handles each packet within one publisher period never queues on arrivals, however bursty the market; there one thread is best. Second, above that period a queueing tail appears, driven by the timing of transactions, not by packet rate or size, and two threads can be better than one: splitting the servicing chain into two stages on separate threads removes most of the tail at the cost of one hop on the median. Third, only the slowest stage matters, so a split pays only if it shortens it. Fourth, just under the period, where the production receiver runs, the remaining tail comes from multi-message packets and variable service times, and the levers are cost per message and spread of service, not thread count. An analytic framework, a burst-limit throughput identity and an exact reduction of the tandem to a single bottleneck server, supports these results.

Comment
123 pages, 36 figures, 31 tables. Code in the Kaspar-HFT repository (https://github.com/vincent212/kaspar-hft)

Closest papers

Nearest neighbours by meaning across the whole index.

Adapting the Actor Model of Concurrency for High-Frequency Trading: Synchronous Message Delivery (fast_send) and a Tick-to-Book Latency Study

cs.DC q-fin.TR ·2026-09-18 ·Vincent Maciejewski
code/data similarity 0.60
Abstract · 31 pages, 4 figures. Companion paper on the market-data arrival process in preparation

The actor model - state isolation, data-race freedom, and sequential single-message reasoning - has long been dismissed as unsuitable for high-frequency trading (HFT): actors seem to imply many threads, a mailbox per actor, and a heap-allocated message plus a context switch per interaction, overhead incompatible with a microsecond budget. This paper argues the dismissal is wrong for co-located actors, with a deployed, measured implementation: kaspar-hft, an open-source C++20 framework. Four extensions adapt the model for HFT: fast_send, a synchronous delivery mechanism in which the sending thread runs the receiver's handler inline and returns the reply as a value; actor groups, which co-schedule actors on one thread behind a shared mailbox; per-actor selectable mailbox queues; and a memory pool. fast_send has receiver transparency: the handler is written identically for synchronous and asynchronous delivery and does not depend on which was used or which thread runs it. A grouped synchronous chain runs on one thread, cutting scheduler context switches from O(N) to O(1); a thread-local call-chain test detects cyclic invocation on one thread before any lock is taken, while a cycle spread across threads deadlocks. Microbenchmarks put the synchronous round trip at tens of nanoseconds. On a live CME market-data feed (ES, NQ, ZN futures), when the socket-reader thread decodes each packet and updates the book itself, socket-to-book medians for book updates are 0.8-1.1 microseconds, and a fast_send hop is about 1% of that. The shared-queue group also yields a production/simulation duality: the same actor code runs unchanged in live trading and deterministic backtest.

0.39

Invisible Yet Dominant: Big Stalls of Kernel I/O Mechanisms in Cloud OLTP Databases

cs.DB cs.OS ·2026-09-11 ·Mitsumasa Kondo
published similarity 0.56
Abstract · Poster presented at SOSP 2026. Companion to arXiv:2603.29052 (SteelDB)

Most databases, including PostgreSQL, RocksDB, and recent AI KV-cache middleware, rely on buffered I/O, delegating write-back to the Linux kernel. On the distributed block storage standard in the cloud, this delegation inherits a hidden bottleneck: each device is drained by a single kernel flusher thread over a high-latency, shallow-queue path. When the drain falls behind, dirty throttling pauses write() system calls, and even reads that must evict dirty pages stall. These stalls are invisible to iostat and every standard counter. This poster observes the stall from inside the kernel, using the multi-volume data placement proposed in SteelDB as the experimental lever. eBPF probes on writeback and block tracepoints separate write-back by issuing context and count every throttle pause. Across three configurations with identical provisioned IOPS and bandwidth but 1, 2, and 4 devices, we show that adding drains, not bandwidth, cuts throttle pauses by 70%, reduces maximum transaction latency by 59%, and raises throughput by 23%.

0.27

Quantifying Observable High-Frequency Swapping on Arbitrum

cs.CR cs.CE ·2026-09-13 ·Shijian Chen, Ya Chen, Jing Cai, Catherine Liu
survey similarity 0.55
Abstract

Layer-2 rollups have reshaped Ethereum's transaction economy by replacing mempool competition with deterministic sequencing, sub-second block times, and negligible gas fees. While these properties suppress classical Miner/Maximal Extractable Value (MEV), they give rise to a new and previously unrecognized behavioral regime. In this paper, we introduce the concept of High-Frequency Swapping (HFS), referring to continuous single-hop swaps at machine cadence within decentralized exchanges. Despite its growing footprint, this phenomenon has not been systematically identified or quantified in prior work. We conduct a large-scale measurement of HFS on Arbitrum, using 18 months of full-chain data (Jan. 2023--Jun. 2024). Specifically, we construct a reproducible pipeline that isolates voluntary single-swap transactions, attributes them to the swapper level, and classifies HFS behavior through inter-arrival dynamics. The analysis uncovers 477 distinct HFS swappers responsible for nearly 30 million swaps and over $10^{11}$ USD in notional volume. Our study conducts a comprehensive empirical investigation of HFS from multiple perspectives. We begin with a global overview of activity patterns, and then examine temporal dynamics, swapper identity, token coverage, and venue concentration, and explore swap size, time gap, and order direction. Finally, we explore case-level behavior, including stablecoin arbitrage, CEX-DEX execution gaps, and short-horizon round-trip trading. Across these dimensions, we identify consistent structural regularities that distinguish HFS from conventional retail or arbitrage activity. This work provides a large-scale address-level empirical characterization of High-Frequency Swapping on Arbitrum and a broad empirical foundation for future research on decentralized market microstructure.

0.26

Completion-Path Credits: Multi-Resource Control for Scale-Up Fabrics

cs.NI ·2026-08-18 ·Fan Yang, Jiaqi Liu, Tao Jiang, Zhan Wang
similarity 0.55
Abstract · 10 pages, 5 figures

Scale-up fabrics connecting GPUs and AI accelerators carry tensor transfers together with remote reads, writes, atomics, and notifications over shared target-side receiver resources. Byte-denominated credits protect link buffers and streaming HBM traffic, but poorly represent small operations dominated by Atomic execution or response injection. This paper presents SemaCredit, a receiver controller that admits each remote-memory operation against a vector of target-resource demands and returns each component when its corresponding HBM, Atomic, or response stage completes. In a deterministic event simulator with multipath queues, eight HBM partitions, a serialized Atomic engine, and a response engine, SemaCredit matches a strong per-resource byte baseline on HBM-hotspot traffic while reducing small-operation P99 latency by 52.4% under Atomic contention and 10.2% under response incast. Application-shaped mixes show 57.7% and 14.5% P99 latency improvements for AllReduce-shaped and remote-read-shaped traffic while matching byte credits on HBM-dominated MoE traffic.

0.12

CheckCo: Checkpoint-Coordinated Load Balancing for Skewed Event Streams at Scale

cs.DC ·2026-11-17 ·Muhammad Anis Uddin Nasir, Reza Sadraei
similarity 0.54
Abstract · This paper is submitted to ICDE 2027 Industry and Application Track

Event streams are rarely uniform: a few dominant keys can generate orders of magnitude more events than the rest, overloading the workers that receive them and capping throughput. We present CheckCo, a checkpoint-coordinated load-balancing algorithm built on a dynamic oracle that estimates per-key frequencies from the live stream, with no prior statistics. Workers count events per key locally; at each checkpoint barrier, the counts are merged into a partitioning function that divides each key into a number of splits roughly proportional to its estimated frequency and packs the splits onto workers, and this function routes the next checkpoint window. We derive approximation bounds on the resulting load imbalance from classical scheduling results and bound its total key fan-out. On a production pipeline processing over 100 billion events per day, the most loaded worker receives only 9.8% more events than the average over a day (imbalance ratio 1.10), against 8.03 for key-based routing in simulation; in simulation, CheckCo also needs less than half the fan-out of earlier partitioners that reach comparable balance. Shifts in the key distribution are reflected in routing from the next checkpoint on, and the load stays balanced through job restarts.

0.17

Causal Overtaking in Fair Transaction Ordering

cs.CR ·2026-11-19 ·Haozhe Feng, Di Wu, Jian Liu
similarity 0.54
Abstract · 22 pages

Fair transaction ordering protocols aim to mitigate front-running using replica-local arrival orders, but prior work often assumes a single submission origin for each responsive transaction. We identify causal overtaking, in which corrupted replicas observe a target and submit identical signed copies of a responsive transaction through multiple client entries. The copies' first arrivals at different replicas can collectively provide ordering support that no individual path achieves, allowing the response to execute before its target. We formalize a class of receipt-order protocols, including Themis, characterize the timing and network conditions under which the attack is feasible, and show that delayed gossip does not rule it out. We further prove that, under our network model, histories requiring opposite transaction orders can produce identical receipt evidence. Thus, receipt-only mechanisms cannot always preserve earlier-submission priority while preventing responsive overtaking. We propose a mitigation and analyze its assumptions and limitations. Finally, we evaluate the attack and mitigation on Themis through simulations driven by AWS and WonderProxy latency measurements.

0.39

Cap-and-Track: Optimizing LLM Inference Request Dispatch Across Shifting Capacity

cs.LG ·2026-11-18 ·Benjamin Novotny, Doug Szeto, Chang Wang, Shicheng Chen, George Zhihua Yuan +4 more
similarity 0.54
Abstract · 20 pages, 6 figures, 2 tables. Includes appendices

A fixed dispatch rate for LLM inference becomes stale as serving fleets resize and request sizes change. Generative output length is unknown at admission, and shared provider pools can lose capacity. We present Cap-and-Track, a design for asynchronous inference across provider capacity and internal GPUs. Meta's traffic shaper implements Cap-and-Track for post and live-video understanding. Refusals signal the backend's current admission ceiling. Durable priority queues hold excess work outside execution, including refused requests for retry. Cap is a fail-fast admission check at the saturated resource that refuses work before a long wait. Using only those refusals, the Track dispatcher probes upward and backs off. It follows the ceiling without output-length prediction or a fleet-level rate to retune. In a concurrent same-traffic production A/B, the shaper ran per-replica Cap plus Track. It served a classifier on 43% fewer H100s than its queued, fixed-rate precursor, though lower-priority work waited longer at backfill peaks. In steady-capacity real-engine replay, Track ran within 1% of the goodput of a fixed rate tuned above measured capacity, with about a quarter of that rate's refusal share. The shaper carries more than 300 million online requests and hundreds of terabytes of media a day. During a backfill, it handled 1.1 billion requests in its busiest 24 hours. Lessons include keeping queues inside inference engines just long enough to fill batches, steering dispatch by capacity refusals alone, keeping dispatch stateful, and sizing fleets by how long low-priority work can wait.

0.28

On Latency-Optimal Implementations of the Timestamp Object

cs.DC ·2026-10-23 ·Aurojit Panda, Scott Shenker, Sam Toueg, Wen Zhang
similarity 0.54
Abstract

Timestamps are widely used in distributed algorithms to order operations. In this paper, we study the timestamp object (TS), which provides a single operation that returns strictly increasing integers. We ask two questions: What is the maximum resilience and the minimum latency of TS implementations in asynchronous message-passing systems, where latency is measured by worst-case message delay? We answer these questions under various types of failure: crash failures, and Byzantine failures with and without digital signatures. To do so, we give lower bounds and algorithms that determine (1) the optimal resilience of TS implementations under each type of failure, and (2) the optimal latency in all cases but one. Along the way, we introduce a new lower-bound proof technique called a cyclic chain argument, which constructs a cycle of indistinguishable schedules along which the returned timestamps must strictly increase, yielding a contradiction.

0.19