CS Paper Radar

Towards Simple Models of Complex SmartNICs

Robert Chang, Teng Jiang, Wonsup Yoon, Daehyeok Kim, Sam Kumar, George Varghese

cs.DC cs.NI ·submitted 2026-09-25 22:34 UTC · v1

Abstract

Cloud vendors push ambitious in-network processing (e.g., crypto, telemetry) onto the NIC to offload servers even as link rates climb to terabit speeds. Vendors have responded with heterogeneous SmartNICs. For example, NVIDIA BlueField-3 interposes---between the wire and the host CPUs---a line-rate eSwitch, a multithreaded Data-Path Accelerator, general-purpose ARM cores, and a sea of fixed-function accelerators. These devices are notoriously hard to program, and harder still to predict. Applications can be implemented in many ways, with each choice potentially hitting a different bottleneck. A designer ideally needs to know---cheaply, and before a line of code is written---feasible choices and their bottlenecks, and design patterns to improve performance. Our paper offers a starting point to answer these questions using what we call the ZRAM model. It pairs a platform graph of processing zones and their channels with a program graph of tasks and their traffic fractions. The application designer or a compiler chooses a placement that maps the program graph onto the platform graph. Three metrics computed directly from this mapping---capability, roofline, and capacity---score the placement, deciding its feasibility and naming the bottleneck resource. We use a DDoS detector as a primary case study, and briefly explore two other applications, decision-tree inference and RDMA traversal. We distill seven design patterns for programming SmartNICs including a key one we call sifting. ZRAM generalizes to other SmartNICs such as Intel IPU E2200 and AMD Pensando Salina 400, and opens a new research agenda that includes compilers and hardware design.

Closest papers

Nearest neighbours by meaning across the whole index.

Balancing Generality and Specialization: A Survey on AI Datacenter Hardware Architecture

cs.AR ·2026-09-21 ·Yufeng Gu, Jiazhen Wang, Reetuparna Das
surveycode/data similarity 0.68
Abstract

Rapidly growing AI workloads are driving large investments in AI datacenters. This survey classifies industrial AI accelerators into four architectural categories and compares their compute and memory organizations. It examines how node-, rack-, and pod-scale interconnects support collective communication, and traces architectural evolution across accelerator generations. The analysis connects advances in arithmetic throughput with changes in precision, data delivery, execution coordination, communication, power delivery, and cooling. It also discusses future design challenges arising from workload diversity, data movement, infrastructure constraints, and model evolution, showing how the trade-off between generality and specialization extends from individual accelerators to datacenter-scale systems. GitHub: github.com/Yufeng98/AI-datacenter

0.21

Xronos: Heterogeneity-Aware Tensor Parallelism for Collaborative LLM Fine-Tuning on Edge CPUs

cs.DC ·2026-09-17 ·Wonmi Choi, Sunjae Park, Dohyeok Kwon, Zhixiong Niu, Yeonho Yoo +2 more
benchmark similarity 0.65
Abstract

Collaborative fine-tuning on edge devices adapts large language models to domain-specific data while keeping each device's data local. State-of-the-art (SOTA) collaborative fine-tuning techniques are largely designed for GPU-based edge devices and rely on pipeline parallelism (PP). However, many edge platforms, including IoT gateways, smart-home hubs, and in-vehicle computers, are primarily CPU-based. This paper reports that PP is ineffective on CPU-based edge devices because the same CPU handles both model computation and communication, which causes severe CPU contention. Our analysis shows that this leads to 5.75$\times$ higher computation stall ratios than on GPU devices on average. Tensor parallelism (TP) can alleviate this contention by separating computation and communication, but existing TP techniques assume homogeneous devices. On heterogeneous CPU edge devices, we find that this assumption causes faster workers to remain idle for up to 34% while waiting for slower devices at synchronization points. To address the limitations, we propose Xronos, a collaborative fine-tuning framework for heterogeneous CPU edge devices. Xronos uses TP as its execution backbone and combines lightweight profiling with heterogeneity-aware tensor partitioning to reduce the straggler bottleneck. Across diverse devices, models, and benchmark tasks, Xronos reduces fine-tuning time by 18% (TP) to 56% (PP) and the ratio of device idle time by $\sim$5.9$\times$ over SOTA techniques, while maintaining the accuracy.

0.21

Operating System Support for Accelerators as First-Class Compute Elements in Heterogeneous Systems

cs.AR cs.DCcs.OS ·2026-10-24 ·Mohammad Shahidzadeh, Lesley Shannon
similarity 0.65
Abstract · 13 pages, 12 figures

Heterogeneous systems-on-chip (SoCs) increasingly rely on domain-specific hardware accelerators (HAs) to deliver performance and energy efficiency gains beyond what general-purpose processors can provide. However, operating systems (O/S), such as Linux, assume that processors are the Compute Elements (CEs) and HAs are peripheral devices rather than specialized CEs. As a result, HA execution incurs substantial software (SW) overheads that reduce the effective benefits of hardware (HW) acceleration. This work demonstrates that significant inefficiencies arise from ``O/S peripheral device taxes'' (O/S taxes), including SW stack-accelerator communication, invocation, interrupt handling, and scheduling. We built MEERKAT, a Field-Programmable Gate Array (FPGA)-based framework for configurable multicore systems that include HAs; it allows the user to migrate Linux O/S support from SW to HW and identify, study, and quantify the impact. Here we study four major sources of O/S tax and introduce techniques to mitigate them. These include a virtual-memory-aware (VMA) communication method that enables efficient HA access to application data, a lightweight HA invocation interface, Interrupt Policy Adaptation (IPA) for dynamic polling and interrupt selection, and an HA-aware scheduler that dynamically distributes work across Central Processing Units (CPUs) and multiple HA instances. Experimental results demonstrate up to $260\times$ faster communication and $2$--$6\times$ performance improvements in wall-clock time over conventional schedulers. Furthermore, our study reveals that overall system performance depends not only on HA speed, but also on O/S overheads, resource availability, and scheduling decisions. These findings highlight the importance of treating HAs as first-class computational resources and motivate a more accelerator-centric O/S design for future heterogeneous systems.

0.15

Zepp: Accelerating Distributed MoE Serving under Relaxed Balance Constraints

cs.DC ·2026-10-08 ·Chang Chen, Andrew Yang, Tiancheng Chen, Jiangfei Duan, Xinwei Qiang +3 more
similarity 0.63
Abstract

As Mixture-of-Experts (MoE) models continue to scale, serving them increasingly relies on expert parallelism (EP) across a growing number of devices. Yet skewed expert workloads create imbalance across computation, communication, and memory, making load balancing a central optimization objective in distributed MoE serving. We observe that balance is not free: operations introduced to balance one dimension can themselves be expensive or imbalanced. This motivates us to rethink balance as a constraint rather than an optimization objective. We present Zepp, which directly optimizes the bottleneck communication in distributed MoE serving subject to simplified balance constraints on physical resources, i.e., GPUs and NICs. Zepp progressively optimizes inter-node communication across placement, routing, and execution. It first places expert replicas to reduce token communication under GPU constraints, then reshapes communication flows through split and merge primitives under NIC constraints, and finally partitions and schedules ex- pert computation to overlap the resulting communication. To adapt to dynamic workloads, Zepp jointly coordinates computation, token communication, and expert-weight movement at each iteration. Together, these designs allow Zepp to pursue the most efficient execution rather than a single-dimension balanced one. We implement Zepp and evaluate it against 7 state-of-the-art MoE serving systems, achieving up to 6.68$\times$ MoE layer speedup and a geometric mean speedup of 1.86$\times$ over the fastest competing baseline.

0.17

Provisioning Networks for AI Supercomputers: A Trace-Driven Study of Performance Sensitivity at Unprecedented Scale

cs.NI ·2026-10-30 ·Tommaso Bonato, Amirreza Rastegari, Daniele De Sensi, Timo Schneider, Siyuan Shen +3 more
publishedtheory similarity 0.63
Abstract · To appear in the proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC 2026)

AI workloads on supercomputers make interconnect performance a primary design constraint. Hardware specifications and microbenchmarks alone do not show how real workloads respond to latency, bandwidth, and jitter. We present a trace-driven study of network sensitivity for training and inference workloads at scales up to 4,096 GPUs. Our methodology combines signature-based trace decomposition with linear programming to make broad network-parameter sweeps feasible. It preserves the recorded NCCL schedules and program dependencies, and estimates a lower bound on runtime under the LogGP model. We validate the compositional analysis against monolithic linear programs, LogGOPSim, and hardware measurements at feasible scales. The resulting latency-bandwidth envelopes identify low-sensitivity regions and performance cliffs that vary substantially across workloads. They also show when bandwidth upgrades provide meaningful returns and when additional network performance has little effect. These results translate execution traces into practical guidance for evaluating network configurations while making the model assumptions explicit.

0.39

Accelerator Choice Is Not Enough: AlphaFold2 Inference on Cloud TPUs

cs.DC cs.LGcs.PF ·2026-09-28 ·Lorenzo Pazienza, Ihab El Bani
code/data similarity 0.63
Abstract · 22 pages, 5 figures, 4 tables. Code and data: https://github.com/lorenzopazienza/alphafold-tpu-benchmark

AlphaFold2 is written in JAX, so the same inference code compiles and runs unchanged on CPUs, GPUs and Google Cloud TPUs. That portability makes the accelerator look like the main decision a user has to make. We show that it is not. Running one AlphaFold2 inference workload across a Colab CPU runtime, an NVIDIA T4 GPU and a dedicated eight-chip Cloud TPU v5e slice, we find a large hardware advantage for the TPU, 0.47 s per call in steady state on a single chip against 13.1 s on the T4 in the same measurement campaign, and three ways in which the software layer decides how much of it a user actually gets. The default execution path uses one chip of the eight, and at list prices the idle capacity makes the slice cost about as much per prediction as the GPU. Batching with JAX's vmap never exceeds single-query throughput, while mapping queries across chips with JAX's pmap gives eight chips 6.5-7.9x the throughput of one on a matched grid; automatic sharding leaves the per-chip footprint unchanged, consistent with replication, most plausibly because AlphaFold2 carries no sharding annotations. Our retained trace analysis of a first call at a new input shape reports about three quarters of the traced span in JAX tracing and compilation rather than execution. Reruns five weeks later reproduced neither cloud baseline, the GPU one off by roughly a factor of two, so the hardware ratio above is specific to one campaign.

0.41

Pipeline-Native Transformers: Co-Designing Model Architecture and CPU Inference for Bandwidth-Efficient Autoregressive Decode

cs.AR cs.LGcs.PF ·2026-08-24 ·Tom Poperszky
code/data similarity 0.63
Abstract · 77 pages, 2 figures. Model weights and binaries: https://huggingface.co/tompoper/cflow

Single-token autoregressive decode on CPUs is bound by memory bandwidth, not arithmetic: a modern CPU sustains roughly 1 TFLOP/s of compute but only about 50 GB/s from main memory, and each generated token must stream every active weight once. This report argues that the most effective response is to co-design the model architecture and the inference runtime together. It presents cflow, a CPU-first streaming engine, alongside a family of pipeline-native transformer architectures whose inter-layer dependency graphs are constructed to permit a vertical, stage-major execution schedule. cflow stores weights as L2-sized tiles in compute-consumption order, reads only the top-k experts of each mixture-of-experts layer, fuses projections, and executes a delay-aware schedule from per-model dependency parameters. Across five architectures trained on TinyStories, one (arch2_4_combined) achieves a 2.00x reduction in critical-path weight bandwidth (9.00 to 4.50 MB/token) within 0.24 perplexity of the best candidate, and the tile layout incurs 7.29x fewer L1-data read misses than a row-major baseline. On a 30.9-billion-parameter pipeline-native MoE, cflow decodes at 5.94 tokens/s (tok/s) on a 32-vCPU Ice Lake server, ahead of llama.cpp (4.75) and the vLLM CPU backend (1.65) on comparably sized dense models. Realizing the expert-delay window as asynchronous I/O overlap on a disk-resident expert tier yields a further net win of up to 1.68x, matching the overlap model within 1%. Measurement refutes one of the eight design claims and leaves a second inconclusive; both are reported in full, with the conditions under which they would hold.

0.26

When Does Distributed AI Inference Need More Wide-Area Bandwidth? A Co-Design Evaluation of Optical, Packet, and Software Levers

cs.DC cs.NI ·2026-08-15 ·Prasanna C
similarity 0.63
Abstract · 10 pages, 4 figures, 3 tables

Wide-area bandwidth per unit of GPU compute falls every hardware generation: in compute-intensity-ratio terms (CIR, bytes per FLOP), the gap between on-package memory and the conventional WAN is four to five orders of magnitude, widening at roughly 12-19% per year. Position papers - including our own - argued this makes elastic optical wide-area capacity necessary for cross-site AI inference. Reviewers correctly objected that such arguments show more bandwidth helps, not that it beats the alternatives: KV recomputation, cache compression, locality-aware routing, scheduling, or an overprovisioned packet backbone. This paper does the comparison. We derive a workload model predicting when moving inference state across sites beats recomputing it - a context-independent crossover at 74-111 Gbps per stream for a 70B multi-head-attention model, falling to 9-14 Gbps under grouped-query attention at 1/8 KV heads - and quantify five sensitivity axes: context length, attention architecture, queueing, agentic compounding, and loss/jitter-induced bandwidth collapse. On economics: at list GPU prices recomputation is cheaper; transfer wins when GPU scarcity and KV reuse multiply effective GPU cost by roughly 5-20x, and modern attention moves the breakeven an order of magnitude in transfer's favour. We position the network levers correctly rather than adversarially: packet networks allocate lit capacity at millisecond timescales; optical fungibility changes how much capacity is lit, at minute timescales, substituting for overprovisioning economically rather than functionally. Finally we specify a ten-metric measurement plan on a three-site production-fibre testbed, framed as an open ecosystem exercise: no single company can - or should - assemble this evidence alone. Every claim is bounded by the regime in which it holds; several findings weaken the naive version of our own thesis, and we state them.

0.10